{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "98bece14",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Execute this cell to install dependencies\n",
    "%pip install sf-hamilton[visualization]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0633cdc",
   "metadata": {},
   "source": [
    "# Simple Single Document Processing Pipeline Example [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dagworks-inc/hamilton/blob/main/examples/LLM_Workflows/RAG_document_extract_chunk_embed/simple_pipeline.ipynb) [![GitHub badge](https://img.shields.io/badge/github-view_source-2b3137?logo=github)](https://github.com/apache/hamilton/blob/main/examples/LLM_Workflows/RAG_document_extract_chunk_embed/simple_pipeline.ipynb)\n",
    "\n",
    "\n",
    "This notebook shows an explains a simple document processing pipeline to feed into a RAG use case. \n",
    "\n",
    "The use case we'll be using is ingesting technical documentation to build a chat bot that can help users get started with the [Hamilton library](https://github.com/apache/hamilton).\n",
    "\n",
    "Specifically, the purpose of this notebook is to help walkthrough and ground the following steps in code using the above use case:\n",
    "\n",
    "![pipeline](pipeline-steps.png)\n",
    "\n",
    "## Libraries Used\n",
    "This code is meant to serve as a blueprint and enable you to change the internal implementations of functions easily for your use case.\n",
    "\n",
    "It uses:\n",
    "* [Hamilton](https://github.com/apache/hamilton), a library to orchestrate, visualize, and keep your code modular. We just have to [name our python functions](https://hamilton.apache.org/concepts/node) appropriately to use it.\n",
    "* [OpenAI](https://www.openai.com) to create embeddings.\n",
    "* [LangChain](https://www.langchain.com) to help chunk text.\n",
    "\n",
    "### Cell Magic explanation\n",
    "You'll see `%%incr_cell_to_module` used below. This is a \"cell magic\". See this [blog post](https://blog.dagworks.io/p/using-ipython-jupyter-magic-commands) for an explanation of them. Here we use it to draw the pipeline as we're building it and then creating a module to group the code to execute the pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3645e902",
   "metadata": {},
   "outputs": [],
   "source": [
    "# install the required packages to run this notebook\n",
    "!pip install sf-hamilton[visualization] openai langchain langchain-core requests pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "099304fd",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-03T22:35:14.243452Z",
     "start_time": "2024-05-03T22:35:14.231603Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/stefankrawczyk/.pyenv/versions/knowledge_retrieval-py39/lib/python3.9/site-packages/pyspark/pandas/__init__.py:50: UserWarning: 'PYARROW_IGNORE_TIMEZONE' environment variable was not set. It is required to set this environment variable to '1' in both driver and executor sides if you use pyarrow>=2.0.0. pandas-on-Spark will set it for you but it does not work if there is a Spark context already launched.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "# We use Hamilton's jupyter magic to help us visualize what we're building\n",
    "%load_ext hamilton.plugins.jupyter_magic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56ae1f3e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# set OPENAI_API_KEY if you haven't\n",
    "import os\n",
    "os.environ[\"OPENAI_API_KEY\"] = \"MY_KEY\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5869a84f",
   "metadata": {},
   "source": [
    "## Documents & Chunks\n",
    "Below we establish some classes to help us describe what we want at the end of this process before we store things.\n",
    "\n",
    "Specifically, what a `Chunk` is and what a `Document` is."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "477e5c68",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-03T22:36:37.957650Z",
     "start_time": "2024-05-03T22:36:37.913299Z"
    },
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 10.0.1 (20240210.2158)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"8pt\" height=\"8pt\"\n",
       " viewBox=\"0.00 0.00 8.00 8.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 4)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-4 4,-4 4,4 -4,4\"/>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x150988d00>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%incr_cell_to_module doc_pipeline 1 --display\n",
    "\n",
    "from typing import NamedTuple, Optional\n",
    "\n",
    "class Chunk(NamedTuple):\n",
    "    \"\"\"Chunk of a Document - text & embedding.\"\"\"\n",
    "    index: int\n",
    "    document_id: str\n",
    "    text: str\n",
    "    embedding: Optional[list[float]]\n",
    "    metadata: Optional[dict[str, str]]\n",
    "    \n",
    "    def add_embedding(self, embedding: list[float]) -> \"Chunk\":\n",
    "        \"\"\"Required to update chunk with embeddings\"\"\"\n",
    "        return Chunk(self.index, self.document_id, self.text, embedding, self.metadata)\n",
    "    \n",
    "    def add_metadata(self, metadata: dict[str, str]) -> \"Chunk\":\n",
    "        \"\"\"Required to update chunk with metadata\"\"\"\n",
    "        return Chunk(self.index, self.document_id, self.text, self.embedding, metadata)\n",
    "\n",
    "class Document(NamedTuple):\n",
    "    \"\"\"Document containing a full raw text, along with pointers to chunks.\"\"\"\n",
    "    id: str\n",
    "    url: str\n",
    "    raw_text: str\n",
    "    chunks: Optional[list[Chunk]]\n",
    "    \n",
    "    def add_chunks(self, chunks: list[Chunk]) -> \"Document\":\n",
    "        \"\"\"Required to update the document when Chunks are created\"\"\"\n",
    "        return Document(self.id, self.url, self.raw_text, chunks)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dfd1350b",
   "metadata": {},
   "source": [
    "# 1. & 2. Load & Extract\n",
    "Here we combine loading and extracting.\n",
    "\n",
    "1. Load the document by pulling the HTML from the website.\n",
    "2. Use a regular expression to extract the content from the page that we're interested in.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "562a2d16",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-03T22:36:38.907993Z",
     "start_time": "2024-05-03T22:36:38.512678Z"
    },
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 10.0.1 (20240210.2158)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"253pt\" height=\"282pt\"\n",
       " viewBox=\"0.00 0.00 252.95 282.30\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 278.3)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-278.3 248.95,-278.3 248.95,4 -4,4\"/>\n",
       "<g id=\"clust1\" class=\"cluster\">\n",
       "<title>cluster__legend</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" points=\"8,-136.3 8,-266.3 92.85,-266.3 92.85,-136.3 8,-136.3\"/>\n",
       "<text text-anchor=\"middle\" x=\"50.43\" y=\"-249\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Legend</text>\n",
       "</g>\n",
       "<!-- raw_document -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>raw_document</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M232.95,-90.1C232.95,-90.1 138.6,-90.1 138.6,-90.1 132.6,-90.1 126.6,-84.1 126.6,-78.1 126.6,-78.1 126.6,-38.5 126.6,-38.5 126.6,-32.5 132.6,-26.5 138.6,-26.5 138.6,-26.5 232.95,-26.5 232.95,-26.5 238.95,-26.5 244.95,-32.5 244.95,-38.5 244.95,-38.5 244.95,-78.1 244.95,-78.1 244.95,-84.1 238.95,-90.1 232.95,-90.1\"/>\n",
       "<text text-anchor=\"start\" x=\"137.4\" y=\"-67\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">raw_document</text>\n",
       "<text text-anchor=\"start\" x=\"154.65\" y=\"-39\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">Document</text>\n",
       "</g>\n",
       "<!-- html_regex -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>html_regex</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M85.6,-126.1C85.6,-126.1 15.25,-126.1 15.25,-126.1 9.25,-126.1 3.25,-120.1 3.25,-114.1 3.25,-114.1 3.25,-74.5 3.25,-74.5 3.25,-68.5 9.25,-62.5 15.25,-62.5 15.25,-62.5 85.6,-62.5 85.6,-62.5 91.6,-62.5 97.6,-68.5 97.6,-74.5 97.6,-74.5 97.6,-114.1 97.6,-114.1 97.6,-120.1 91.6,-126.1 85.6,-126.1\"/>\n",
       "<text text-anchor=\"start\" x=\"14.05\" y=\"-103\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">html_regex</text>\n",
       "<text text-anchor=\"start\" x=\"42.93\" y=\"-75\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- html_regex&#45;&gt;raw_document -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>html_regex&#45;&gt;raw_document</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M98.06,-81.71C103.79,-80.16 109.74,-78.56 115.71,-76.95\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"116.23,-80.43 124.98,-74.44 114.41,-73.67 116.23,-80.43\"/>\n",
       "</g>\n",
       "<!-- _raw_document_inputs -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>_raw_document_inputs</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"83.23,-44.6 17.63,-44.6 17.63,0 83.23,0 83.23,-44.6\"/>\n",
       "<text text-anchor=\"start\" x=\"32.43\" y=\"-16.5\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">url</text>\n",
       "<text text-anchor=\"start\" x=\"53.43\" y=\"-16.5\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- _raw_document_inputs&#45;&gt;raw_document -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>_raw_document_inputs&#45;&gt;raw_document</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M83.71,-31.01C93.3,-33.61 104.21,-36.55 115.22,-39.52\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"114.19,-42.87 124.76,-42.1 116.02,-36.11 114.19,-42.87\"/>\n",
       "</g>\n",
       "<!-- input -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>input</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"77.43,-180.6 23.43,-180.6 23.43,-144 77.43,-144 77.43,-180.6\"/>\n",
       "<text text-anchor=\"middle\" x=\"50.43\" y=\"-156.5\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">input</text>\n",
       "</g>\n",
       "<!-- function -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>function</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M72.85,-235.6C72.85,-235.6 28,-235.6 28,-235.6 22,-235.6 16,-229.6 16,-223.6 16,-223.6 16,-211 16,-211 16,-205 22,-199 28,-199 28,-199 72.85,-199 72.85,-199 78.85,-199 84.85,-205 84.85,-211 84.85,-211 84.85,-223.6 84.85,-223.6 84.85,-229.6 78.85,-235.6 72.85,-235.6\"/>\n",
       "<text text-anchor=\"middle\" x=\"50.43\" y=\"-211.5\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">function</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x150988df0>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%incr_cell_to_module doc_pipeline 2 --display\n",
    "import requests \n",
    "import re\n",
    "import uuid\n",
    "\n",
    "def html_regex() -> str:\n",
    "    \"\"\"Context dependent logic for getting the right part of the HTML document.\"\"\"\n",
    "    return r'<article role=\"main\" id=\"furo-main-content\">(.*?)</article>'\n",
    "\n",
    "def raw_document(url: str, html_regex: str) -> Document:\n",
    "    \"\"\"Loads and parses the HTML from a URL, returning the html text of interest.\n",
    "\n",
    "    :param url: the url to pull.\n",
    "    :param html_regex: the regext to use to get the contents out of.\n",
    "    :return: sub-portion of the HTML\n",
    "    \"\"\"\n",
    "    html_text = requests.get(url).text\n",
    "    article = re.findall(html_regex, html_text, re.DOTALL)\n",
    "    if not article:\n",
    "        raise ValueError(f\"No article found in {url}\")\n",
    "    raw_text = article[0].strip()\n",
    "    return Document(str(uuid.uuid4()), url, raw_text, None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "d67e6137",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-03T23:25:08.180619Z",
     "start_time": "2024-05-03T23:25:08.102387Z"
    },
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Document(id='ff0d191c-e3a6-4fb4-bade-a0d58ddfaebb', url='https://hamilton.dagworks.io/en/latest/concepts/best-practices/code-organization/', raw_text='<section id=\"code-organization\">\\n<h1>Code Organization<a class=\"headerlink\" href=\"#code-organization\" title=\"Link to this heading\">¶</a></h1>\\n<p>Hamilton will force you to organize your code! Here’s some tips.</p>\\n<p>Hamilton forces you to put your code into modules that are distinct from where you run your code.</p>\\n<p>You’ll soon find that a single python module does not make sense, and so you’ll organically start to (very likely) put\\nlike functions with like functions, i.e. thus creating domain specific modules –&gt; <cite>use this to your development\\nadvantage!</cite></p>\\n<p>At Stitch Fix we:</p>\\n<ol class=\"arabic simple\">\\n<li><p>Use modules to model team thinking, e.g. date_features.py.</p></li>\\n<li><p>Use modules to helps isolate what you’re working on.</p></li>\\n<li><p>Use modules to replace parts of your Hamilton dataflow very easily for different contexts.</p></li>\\n</ol>\\n<section id=\"team-thinking\">\\n<h2>Team thinking<a class=\"headerlink\" href=\"#team-thinking\" title=\"Link to this heading\">¶</a></h2>\\n<p>You’ll need to curate your modules. We suggest orienting this around how teams think about the business.</p>\\n<p>E.g. marketing spend features should be in the same module, or in separate modules but in the same directory/package.</p>\\n<p>This will then make it easy for people to browse the code base and discover what is available.</p>\\n</section>\\n<section id=\"helps-isolate-what-you-re-working-on\">\\n<h2>Helps isolate what you’re working on<a class=\"headerlink\" href=\"#helps-isolate-what-you-re-working-on\" title=\"Link to this heading\">¶</a></h2>\\n<p>Grouping functions into modules then helps set the tone for what you’re working on. It helps set the “namespace”, if\\nyou will, for that function. Thus you can have the same function name used in multiple modules, as long as only one of\\nthose modules is imported to build the DAG.</p>\\n<p>Thus modules help you create boundaries in your code base to isolate functions that you’ll want to change inputs to.</p>\\n</section>\\n<section id=\"enables-you-to-replace-parts-of-your-dag-easily-for-different-contexts\">\\n<h2>Enables you to replace parts of your DAG easily for different contexts<a class=\"headerlink\" href=\"#enables-you-to-replace-parts-of-your-dag-easily-for-different-contexts\" title=\"Link to this heading\">¶</a></h2>\\n<p>The names you provide as inputs to functions form a defined “interface”, to borrow a computer science term, so if you\\nwant to swap/change/augment an input, having a function that would map to it defined in another module(s) provides a\\nlot of flexibility. Rather than having a single module with all functions defined in it, separating the functions into\\ndifferent modules could be a productivity win.</p>\\n<p>Why? That’s because when you come to tell Hamilton what functions constitute your dataflow (i.e. DAG), you’ll be able\\nto simply replace/add/change the module being passed. So if you want to compute inputs for certain functions\\ndifferently, this composability of including/excluding modules, when building the DAG provides a lot of flexibility\\nthat you can exploit to make your development cycle faster.</p>\\n</section>\\n</section>', chunks=None)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# print(requests.get(\"https://hamilton.apache.org/concepts/best-practices/code-organization/\").text)\n",
    "# we can test that this works by running the functions:\n",
    "url = \"https://hamilton.apache.org/concepts/best-practices/code-organization/\"\n",
    "doc_pipeline.raw_document(url, doc_pipeline.html_regex())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98c91735",
   "metadata": {},
   "source": [
    "# 3. Chunk the text into sections\n",
    "\n",
    "Here we use part of LangChain's libraries to split the HTML into chunks that we will individually index and then search for when trying to answer questions. Assuming the documentation is well structured, it probably makes sense to \"chunk\" by headings, as that will logically split the text into (hopefully) self-contained sections (stripping HTML from the text in the process). But, these sections might still be quite large, so we want to further split this text. E.g. going from paragraphs to smaller sections.\n",
    "\n",
    "We then attach the `chunks` to the `document` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "b60633dc",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-03T22:36:40.256694Z",
     "start_time": "2024-05-03T22:36:39.910957Z"
    },
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 10.0.1 (20240210.2158)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"560pt\" height=\"451pt\"\n",
       " viewBox=\"0.00 0.00 560.05 450.80\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 446.8)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-446.8 556.05,-446.8 556.05,4 -4,4\"/>\n",
       "<g id=\"clust1\" class=\"cluster\">\n",
       "<title>cluster__legend</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" points=\"25.88,-304.8 25.88,-434.8 110.72,-434.8 110.72,-304.8 25.88,-304.8\"/>\n",
       "<text text-anchor=\"middle\" x=\"68.3\" y=\"-417.5\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Legend</text>\n",
       "</g>\n",
       "<!-- chunked_document -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>chunked_document</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M540.05,-145.6C540.05,-145.6 414.2,-145.6 414.2,-145.6 408.2,-145.6 402.2,-139.6 402.2,-133.6 402.2,-133.6 402.2,-94 402.2,-94 402.2,-88 408.2,-82 414.2,-82 414.2,-82 540.05,-82 540.05,-82 546.05,-82 552.05,-88 552.05,-94 552.05,-94 552.05,-133.6 552.05,-133.6 552.05,-139.6 546.05,-145.6 540.05,-145.6\"/>\n",
       "<text text-anchor=\"start\" x=\"413\" y=\"-122.5\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">chunked_document</text>\n",
       "<text text-anchor=\"start\" x=\"446\" y=\"-94.5\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">Document</text>\n",
       "</g>\n",
       "<!-- html_regex -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>html_regex</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M103.47,-294.6C103.47,-294.6 33.12,-294.6 33.12,-294.6 27.12,-294.6 21.12,-288.6 21.12,-282.6 21.12,-282.6 21.12,-243 21.12,-243 21.12,-237 27.12,-231 33.12,-231 33.12,-231 103.47,-231 103.47,-231 109.47,-231 115.47,-237 115.47,-243 115.47,-243 115.47,-282.6 115.47,-282.6 115.47,-288.6 109.47,-294.6 103.47,-294.6\"/>\n",
       "<text text-anchor=\"start\" x=\"31.92\" y=\"-271.5\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">html_regex</text>\n",
       "<text text-anchor=\"start\" x=\"60.8\" y=\"-243.5\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- raw_document -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>raw_document</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M316.57,-227.6C316.57,-227.6 222.22,-227.6 222.22,-227.6 216.22,-227.6 210.22,-221.6 210.22,-215.6 210.22,-215.6 210.22,-176 210.22,-176 210.22,-170 216.22,-164 222.22,-164 222.22,-164 316.57,-164 316.57,-164 322.57,-164 328.57,-170 328.57,-176 328.57,-176 328.57,-215.6 328.57,-215.6 328.57,-221.6 322.57,-227.6 316.57,-227.6\"/>\n",
       "<text text-anchor=\"start\" x=\"221.02\" y=\"-204.5\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">raw_document</text>\n",
       "<text text-anchor=\"start\" x=\"238.27\" y=\"-176.5\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">Document</text>\n",
       "</g>\n",
       "<!-- html_regex&#45;&gt;raw_document -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>html_regex&#45;&gt;raw_document</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M115.73,-247.18C140.48,-238.85 171.43,-228.43 198.98,-219.16\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"199.95,-222.53 208.31,-216.02 197.72,-215.89 199.95,-222.53\"/>\n",
       "</g>\n",
       "<!-- raw_document&#45;&gt;chunked_document -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>raw_document&#45;&gt;chunked_document</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M328.8,-172.52C348.21,-164.78 370.24,-156.01 391.23,-147.64\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"392.37,-150.95 400.36,-144 389.78,-144.45 392.37,-150.95\"/>\n",
       "</g>\n",
       "<!-- text_chunker -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>text_chunker</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M361.2,-145.6C361.2,-145.6 177.6,-145.6 177.6,-145.6 171.6,-145.6 165.6,-139.6 165.6,-133.6 165.6,-133.6 165.6,-94 165.6,-94 165.6,-88 171.6,-82 177.6,-82 177.6,-82 361.2,-82 361.2,-82 367.2,-82 373.2,-88 373.2,-94 373.2,-94 373.2,-133.6 373.2,-133.6 373.2,-139.6 367.2,-145.6 361.2,-145.6\"/>\n",
       "<text text-anchor=\"start\" x=\"227.4\" y=\"-122.5\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">text_chunker</text>\n",
       "<text text-anchor=\"start\" x=\"176.4\" y=\"-94.5\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">RecursiveCharacterTextSplitter</text>\n",
       "</g>\n",
       "<!-- text_chunker&#45;&gt;chunked_document -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>text_chunker&#45;&gt;chunked_document</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M373.56,-113.8C379.22,-113.8 384.87,-113.8 390.46,-113.8\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"390.29,-117.3 400.29,-113.8 390.29,-110.3 390.29,-117.3\"/>\n",
       "</g>\n",
       "<!-- html_chunker -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>html_chunker</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M342.45,-63.6C342.45,-63.6 196.35,-63.6 196.35,-63.6 190.35,-63.6 184.35,-57.6 184.35,-51.6 184.35,-51.6 184.35,-12 184.35,-12 184.35,-6 190.35,0 196.35,0 196.35,0 342.45,0 342.45,0 348.45,0 354.45,-6 354.45,-12 354.45,-12 354.45,-51.6 354.45,-51.6 354.45,-57.6 348.45,-63.6 342.45,-63.6\"/>\n",
       "<text text-anchor=\"start\" x=\"224.77\" y=\"-40.5\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">html_chunker</text>\n",
       "<text text-anchor=\"start\" x=\"195.15\" y=\"-12.5\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">HTMLHeaderTextSplitter</text>\n",
       "</g>\n",
       "<!-- html_chunker&#45;&gt;chunked_document -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>html_chunker&#45;&gt;chunked_document</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M351.41,-64.09C364.54,-69.32 378.17,-74.76 391.39,-80.03\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"389.89,-83.2 400.48,-83.65 392.48,-76.69 389.89,-83.2\"/>\n",
       "</g>\n",
       "<!-- _raw_document_inputs -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>_raw_document_inputs</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"101.1,-213.1 35.5,-213.1 35.5,-168.5 101.1,-168.5 101.1,-213.1\"/>\n",
       "<text text-anchor=\"start\" x=\"50.3\" y=\"-185\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">url</text>\n",
       "<text text-anchor=\"start\" x=\"71.3\" y=\"-185\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- _raw_document_inputs&#45;&gt;raw_document -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>_raw_document_inputs&#45;&gt;raw_document</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M101.29,-191.6C127.44,-192.26 165.38,-193.21 198.5,-194.04\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"198.39,-197.54 208.47,-194.3 198.57,-190.55 198.39,-197.54\"/>\n",
       "</g>\n",
       "<!-- _text_chunker_inputs -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>_text_chunker_inputs</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"136.6,-146.6 0,-146.6 0,-81 136.6,-81 136.6,-146.6\"/>\n",
       "<text text-anchor=\"start\" x=\"24.55\" y=\"-118.5\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">chunk_size</text>\n",
       "<text text-anchor=\"start\" x=\"107.67\" y=\"-118.5\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">int</text>\n",
       "<text text-anchor=\"start\" x=\"14.42\" y=\"-97.5\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">chunk_overlap</text>\n",
       "<text text-anchor=\"start\" x=\"107.67\" y=\"-97.5\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">int</text>\n",
       "</g>\n",
       "<!-- _text_chunker_inputs&#45;&gt;text_chunker -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>_text_chunker_inputs&#45;&gt;text_chunker</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M136.98,-113.8C142.55,-113.8 148.26,-113.8 154.04,-113.8\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"153.71,-117.3 163.71,-113.8 153.71,-110.3 153.71,-117.3\"/>\n",
       "</g>\n",
       "<!-- input -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>input</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"95.3,-349.1 41.3,-349.1 41.3,-312.5 95.3,-312.5 95.3,-349.1\"/>\n",
       "<text text-anchor=\"middle\" x=\"68.3\" y=\"-325\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">input</text>\n",
       "</g>\n",
       "<!-- function -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>function</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M90.72,-404.1C90.72,-404.1 45.87,-404.1 45.87,-404.1 39.87,-404.1 33.87,-398.1 33.87,-392.1 33.87,-392.1 33.87,-379.5 33.87,-379.5 33.87,-373.5 39.87,-367.5 45.87,-367.5 45.87,-367.5 90.72,-367.5 90.72,-367.5 96.72,-367.5 102.72,-373.5 102.72,-379.5 102.72,-379.5 102.72,-392.1 102.72,-392.1 102.72,-398.1 96.72,-404.1 90.72,-404.1\"/>\n",
       "<text text-anchor=\"middle\" x=\"68.3\" y=\"-380\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">function</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x150b72a60>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%incr_cell_to_module doc_pipeline 3 --display\n",
    "\n",
    "from langchain import text_splitter\n",
    "\n",
    "def html_chunker() -> text_splitter.HTMLHeaderTextSplitter:\n",
    "    \"\"\"Object to help split HTML into chunks\"\"\"\n",
    "    headers_to_split_on = [\n",
    "        (\"h1\", \"Header 1\"),\n",
    "        (\"h2\", \"Header 2\"),\n",
    "        (\"h3\", \"Header 3\"),\n",
    "    ]\n",
    "    html_chunker = text_splitter.HTMLHeaderTextSplitter(\n",
    "        headers_to_split_on=headers_to_split_on\n",
    "    )\n",
    "    return html_chunker\n",
    "\n",
    "def text_chunker(\n",
    "        chunk_size: int = 256, \n",
    "        chunk_overlap: int = 0) -> text_splitter.RecursiveCharacterTextSplitter:\n",
    "    \"\"\"Object to further split chunks\"\"\"\n",
    "    return text_splitter.RecursiveCharacterTextSplitter(\n",
    "        chunk_size=chunk_size, chunk_overlap=chunk_overlap\n",
    "    )\n",
    "\n",
    "def chunked_document(\n",
    "    raw_document: Document,\n",
    "    html_chunker: text_splitter.HTMLHeaderTextSplitter,\n",
    "    text_chunker: text_splitter.RecursiveCharacterTextSplitter,\n",
    ") -> Document:\n",
    "    \"\"\"This function takes in HTML, chunks the HTML, and then chunks it into text chunks.\"\"\"\n",
    "    header_splits = html_chunker.split_text(raw_document.raw_text)\n",
    "    text_chunks = text_chunker.split_documents(header_splits)\n",
    "    chunks = []\n",
    "    for i, text_chunk in enumerate(text_chunks):\n",
    "        chunks.append(\n",
    "            Chunk(\n",
    "                index=i,\n",
    "                document_id=raw_document.id,\n",
    "                text=text_chunk.page_content,\n",
    "                embedding=None,\n",
    "                metadata=text_chunk.metadata,\n",
    "            )\n",
    "        )\n",
    "    # create new raw_document object\n",
    "    raw_document = raw_document.add_chunks(chunks)\n",
    "    return raw_document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "4443b67f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content='some text  \\nsome more text', metadata={'Header 1': 'title'}),\n",
       " Document(page_content='section text more text', metadata={'Header 1': 'title', 'Header 2': 'subsection1'})]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# example what the HTML chunker is doing:\n",
    "doc_pipeline.html_chunker().split_text(\"<h1>title</h1><p>some text</p><p>some more text</p><h2>subsection1</h2><p>section text<br/> more text</p>\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "49108a56",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-03T22:36:40.549733Z",
     "start_time": "2024-05-03T22:36:40.541836Z"
    },
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['this', 'is', 'some', 'text']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# example what the text chunker is doing\n",
    "doc_pipeline.text_chunker(5, 0).split_text(\"this is some text\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1bfed5e5",
   "metadata": {},
   "source": [
    "For an interactive visualization [this website](https://chunkviz.up.railway.app/#explanation) provides a way for you to understand what is going with the text splitting at least."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ebe8825d",
   "metadata": {},
   "source": [
    "# 4. Embed the text chunks\n",
    "For each text chunk, we send a request to OpenAI to get a vector embedding for that text.\n",
    "This vector of numbers is a \"semantic\" representation in numbers of the context in the text sent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "f4eb1360",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-04T00:41:25.293450Z",
     "start_time": "2024-05-04T00:41:24.812160Z"
    },
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 10.0.1 (20240210.2158)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"752pt\" height=\"451pt\"\n",
       " viewBox=\"0.00 0.00 751.65 450.80\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 446.8)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-446.8 747.65,-446.8 747.65,4 -4,4\"/>\n",
       "<g id=\"clust1\" class=\"cluster\">\n",
       "<title>cluster__legend</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" points=\"25.88,-304.8 25.88,-434.8 110.72,-434.8 110.72,-304.8 25.88,-304.8\"/>\n",
       "<text text-anchor=\"middle\" x=\"68.3\" y=\"-417.5\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Legend</text>\n",
       "</g>\n",
       "<!-- chunked_document -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>chunked_document</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M540.05,-145.6C540.05,-145.6 414.2,-145.6 414.2,-145.6 408.2,-145.6 402.2,-139.6 402.2,-133.6 402.2,-133.6 402.2,-94 402.2,-94 402.2,-88 408.2,-82 414.2,-82 414.2,-82 540.05,-82 540.05,-82 546.05,-82 552.05,-88 552.05,-94 552.05,-94 552.05,-133.6 552.05,-133.6 552.05,-139.6 546.05,-145.6 540.05,-145.6\"/>\n",
       "<text text-anchor=\"start\" x=\"413\" y=\"-122.5\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">chunked_document</text>\n",
       "<text text-anchor=\"start\" x=\"446\" y=\"-94.5\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">Document</text>\n",
       "</g>\n",
       "<!-- embedded_document -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>embedded_document</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M731.65,-186.6C731.65,-186.6 593.05,-186.6 593.05,-186.6 587.05,-186.6 581.05,-180.6 581.05,-174.6 581.05,-174.6 581.05,-135 581.05,-135 581.05,-129 587.05,-123 593.05,-123 593.05,-123 731.65,-123 731.65,-123 737.65,-123 743.65,-129 743.65,-135 743.65,-135 743.65,-174.6 743.65,-174.6 743.65,-180.6 737.65,-186.6 731.65,-186.6\"/>\n",
       "<text text-anchor=\"start\" x=\"591.85\" y=\"-163.5\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">embedded_document</text>\n",
       "<text text-anchor=\"start\" x=\"631.23\" y=\"-135.5\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">Document</text>\n",
       "</g>\n",
       "<!-- chunked_document&#45;&gt;embedded_document -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>chunked_document&#45;&gt;embedded_document</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M552.39,-130.42C558.07,-131.69 563.84,-132.98 569.62,-134.27\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"568.7,-137.65 579.22,-136.42 570.23,-130.82 568.7,-137.65\"/>\n",
       "</g>\n",
       "<!-- html_regex -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>html_regex</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M103.47,-294.6C103.47,-294.6 33.12,-294.6 33.12,-294.6 27.12,-294.6 21.12,-288.6 21.12,-282.6 21.12,-282.6 21.12,-243 21.12,-243 21.12,-237 27.12,-231 33.12,-231 33.12,-231 103.47,-231 103.47,-231 109.47,-231 115.47,-237 115.47,-243 115.47,-243 115.47,-282.6 115.47,-282.6 115.47,-288.6 109.47,-294.6 103.47,-294.6\"/>\n",
       "<text text-anchor=\"start\" x=\"31.92\" y=\"-271.5\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">html_regex</text>\n",
       "<text text-anchor=\"start\" x=\"60.8\" y=\"-243.5\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- raw_document -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>raw_document</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M316.57,-227.6C316.57,-227.6 222.22,-227.6 222.22,-227.6 216.22,-227.6 210.22,-221.6 210.22,-215.6 210.22,-215.6 210.22,-176 210.22,-176 210.22,-170 216.22,-164 222.22,-164 222.22,-164 316.57,-164 316.57,-164 322.57,-164 328.57,-170 328.57,-176 328.57,-176 328.57,-215.6 328.57,-215.6 328.57,-221.6 322.57,-227.6 316.57,-227.6\"/>\n",
       "<text text-anchor=\"start\" x=\"221.02\" y=\"-204.5\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">raw_document</text>\n",
       "<text text-anchor=\"start\" x=\"238.27\" y=\"-176.5\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">Document</text>\n",
       "</g>\n",
       "<!-- html_regex&#45;&gt;raw_document -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>html_regex&#45;&gt;raw_document</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M115.73,-247.18C140.48,-238.85 171.43,-228.43 198.98,-219.16\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"199.95,-222.53 208.31,-216.02 197.72,-215.89 199.95,-222.53\"/>\n",
       "</g>\n",
       "<!-- raw_document&#45;&gt;chunked_document -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>raw_document&#45;&gt;chunked_document</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M328.8,-172.52C348.21,-164.78 370.24,-156.01 391.23,-147.64\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"392.37,-150.95 400.36,-144 389.78,-144.45 392.37,-150.95\"/>\n",
       "</g>\n",
       "<!-- client -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>client</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M499.18,-227.6C499.18,-227.6 455.07,-227.6 455.07,-227.6 449.07,-227.6 443.07,-221.6 443.07,-215.6 443.07,-215.6 443.07,-176 443.07,-176 443.07,-170 449.07,-164 455.07,-164 455.07,-164 499.18,-164 499.18,-164 505.18,-164 511.18,-170 511.18,-176 511.18,-176 511.18,-215.6 511.18,-215.6 511.18,-221.6 505.18,-227.6 499.18,-227.6\"/>\n",
       "<text text-anchor=\"start\" x=\"459.5\" y=\"-204.5\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">client</text>\n",
       "<text text-anchor=\"start\" x=\"453.88\" y=\"-176.5\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">OpenAI</text>\n",
       "</g>\n",
       "<!-- client&#45;&gt;embedded_document -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>client&#45;&gt;embedded_document</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M511.31,-188.38C527.83,-184.68 548.7,-180.01 569.58,-175.33\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"570.35,-178.75 579.34,-173.15 568.82,-171.92 570.35,-178.75\"/>\n",
       "</g>\n",
       "<!-- text_chunker -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>text_chunker</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M361.2,-145.6C361.2,-145.6 177.6,-145.6 177.6,-145.6 171.6,-145.6 165.6,-139.6 165.6,-133.6 165.6,-133.6 165.6,-94 165.6,-94 165.6,-88 171.6,-82 177.6,-82 177.6,-82 361.2,-82 361.2,-82 367.2,-82 373.2,-88 373.2,-94 373.2,-94 373.2,-133.6 373.2,-133.6 373.2,-139.6 367.2,-145.6 361.2,-145.6\"/>\n",
       "<text text-anchor=\"start\" x=\"227.4\" y=\"-122.5\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">text_chunker</text>\n",
       "<text text-anchor=\"start\" x=\"176.4\" y=\"-94.5\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">RecursiveCharacterTextSplitter</text>\n",
       "</g>\n",
       "<!-- text_chunker&#45;&gt;chunked_document -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>text_chunker&#45;&gt;chunked_document</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M373.56,-113.8C379.22,-113.8 384.87,-113.8 390.46,-113.8\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"390.29,-117.3 400.29,-113.8 390.29,-110.3 390.29,-117.3\"/>\n",
       "</g>\n",
       "<!-- html_chunker -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>html_chunker</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M342.45,-63.6C342.45,-63.6 196.35,-63.6 196.35,-63.6 190.35,-63.6 184.35,-57.6 184.35,-51.6 184.35,-51.6 184.35,-12 184.35,-12 184.35,-6 190.35,0 196.35,0 196.35,0 342.45,0 342.45,0 348.45,0 354.45,-6 354.45,-12 354.45,-12 354.45,-51.6 354.45,-51.6 354.45,-57.6 348.45,-63.6 342.45,-63.6\"/>\n",
       "<text text-anchor=\"start\" x=\"224.77\" y=\"-40.5\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">html_chunker</text>\n",
       "<text text-anchor=\"start\" x=\"195.15\" y=\"-12.5\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">HTMLHeaderTextSplitter</text>\n",
       "</g>\n",
       "<!-- html_chunker&#45;&gt;chunked_document -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>html_chunker&#45;&gt;chunked_document</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M351.41,-64.09C364.54,-69.32 378.17,-74.76 391.39,-80.03\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"389.89,-83.2 400.48,-83.65 392.48,-76.69 389.89,-83.2\"/>\n",
       "</g>\n",
       "<!-- _raw_document_inputs -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>_raw_document_inputs</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"101.1,-213.1 35.5,-213.1 35.5,-168.5 101.1,-168.5 101.1,-213.1\"/>\n",
       "<text text-anchor=\"start\" x=\"50.3\" y=\"-185\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">url</text>\n",
       "<text text-anchor=\"start\" x=\"71.3\" y=\"-185\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- _raw_document_inputs&#45;&gt;raw_document -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>_raw_document_inputs&#45;&gt;raw_document</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M101.29,-191.6C127.44,-192.26 165.38,-193.21 198.5,-194.04\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"198.39,-197.54 208.47,-194.3 198.57,-190.55 198.39,-197.54\"/>\n",
       "</g>\n",
       "<!-- _text_chunker_inputs -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>_text_chunker_inputs</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"136.6,-146.6 0,-146.6 0,-81 136.6,-81 136.6,-146.6\"/>\n",
       "<text text-anchor=\"start\" x=\"24.55\" y=\"-118.5\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">chunk_size</text>\n",
       "<text text-anchor=\"start\" x=\"107.67\" y=\"-118.5\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">int</text>\n",
       "<text text-anchor=\"start\" x=\"14.42\" y=\"-97.5\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">chunk_overlap</text>\n",
       "<text text-anchor=\"start\" x=\"107.67\" y=\"-97.5\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">int</text>\n",
       "</g>\n",
       "<!-- _text_chunker_inputs&#45;&gt;text_chunker -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>_text_chunker_inputs&#45;&gt;text_chunker</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M136.98,-113.8C142.55,-113.8 148.26,-113.8 154.04,-113.8\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"153.71,-117.3 163.71,-113.8 153.71,-110.3 153.71,-117.3\"/>\n",
       "</g>\n",
       "<!-- input -->\n",
       "<g id=\"node10\" class=\"node\">\n",
       "<title>input</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"95.3,-349.1 41.3,-349.1 41.3,-312.5 95.3,-312.5 95.3,-349.1\"/>\n",
       "<text text-anchor=\"middle\" x=\"68.3\" y=\"-325\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">input</text>\n",
       "</g>\n",
       "<!-- function -->\n",
       "<g id=\"node11\" class=\"node\">\n",
       "<title>function</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M90.72,-404.1C90.72,-404.1 45.87,-404.1 45.87,-404.1 39.87,-404.1 33.87,-398.1 33.87,-392.1 33.87,-392.1 33.87,-379.5 33.87,-379.5 33.87,-373.5 39.87,-367.5 45.87,-367.5 45.87,-367.5 90.72,-367.5 90.72,-367.5 96.72,-367.5 102.72,-373.5 102.72,-379.5 102.72,-379.5 102.72,-392.1 102.72,-392.1 102.72,-398.1 96.72,-404.1 90.72,-404.1\"/>\n",
       "<text text-anchor=\"middle\" x=\"68.3\" y=\"-380\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">function</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x150dbb880>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%incr_cell_to_module doc_pipeline 4 --display\n",
    "import openai\n",
    "\n",
    "def client() -> openai.OpenAI:\n",
    "    return openai.OpenAI()\n",
    "\n",
    "def embedded_document(\n",
    "    chunked_document: Document,\n",
    "    client: openai.OpenAI,\n",
    ") -> Document:\n",
    "    \"\"\"This function takes in a list of documents and outputs a list of documents with embeddings.\n",
    "\n",
    "    This assumes you have OPENAI_API_KEY in your environment. If not, set it.\n",
    "    \"\"\"\n",
    "    for idx, chunk in enumerate(chunked_document.chunks): \n",
    "        response = client.embeddings.create(input=chunk.text, \n",
    "                                            model=\"text-embedding-3-small\")\n",
    "        chunk = chunk.add_embedding(response.data[0].embedding)\n",
    "        # mutate existing document \n",
    "        chunked_document.chunks[idx] = chunk\n",
    "    return chunked_document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "9428e6e6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[-0.0285369660705328,\n",
       " 0.062304042279720306,\n",
       " -0.01684781350195408,\n",
       " -0.015433112159371376,\n",
       " -0.049328796565532684,\n",
       " -0.0613894909620285,\n",
       " 0.0034045730717480183,\n",
       " 0.007237843703478575,\n",
       " 0.01143907941877842,\n",
       " 0.0047121006064116955,\n",
       " 0.057016775012016296,\n",
       " 0.011289035901427269,\n",
       " -0.04046905040740967,\n",
       " -0.009974363259971142,\n",
       " 0.04890010133385658,\n",
       " 0.004719245713204145,\n",
       " 0.020205944776535034,\n",
       " -0.010210146196186543,\n",
       " -0.01826251484453678,\n",
       " 0.0014370299177244306,\n",
       " -0.0010440571932122111,\n",
       " 0.011581978760659695,\n",
       " 0.0573311522603035,\n",
       " -0.0002672661212272942,\n",
       " 0.005101501010358334,\n",
       " 0.0016808516811579466,\n",
       " 0.009188417345285416,\n",
       " 0.026364900171756744,\n",
       " 0.0593317411839962,\n",
       " -0.03583911433815956,\n",
       " -0.019105620682239532,\n",
       " -0.04095490649342537,\n",
       " -0.010867482982575893,\n",
       " -0.08722566068172455,\n",
       " 0.01771949790418148,\n",
       " 0.012567983008921146,\n",
       " 0.020706091076135635,\n",
       " 0.032123737037181854,\n",
       " 0.0005912453634664416,\n",
       " -0.010953222401440144,\n",
       " -0.0053444295190274715,\n",
       " 0.018919851630926132,\n",
       " 0.038982897996902466,\n",
       " -0.02386416308581829,\n",
       " 0.02349262498319149,\n",
       " 0.031895097345113754,\n",
       " -0.014497121796011925,\n",
       " -0.006133947521448135,\n",
       " -0.025779012590646744,\n",
       " 0.050643470138311386,\n",
       " -0.012782331556081772,\n",
       " 0.019691508263349533,\n",
       " -0.0292657520622015,\n",
       " 0.06601942330598831,\n",
       " -0.011853487230837345,\n",
       " 0.015618880279362202,\n",
       " -0.028165429830551147,\n",
       " 0.01724793203175068,\n",
       " -0.026221999898552895,\n",
       " 0.015361662022769451,\n",
       " 0.0596175380051136,\n",
       " -0.003081263741478324,\n",
       " 0.025250285863876343,\n",
       " -0.04018324986100197,\n",
       " 0.004201236180961132,\n",
       " 0.032781071960926056,\n",
       " -0.06339007616043091,\n",
       " 0.01779094897210598,\n",
       " -0.007723700720816851,\n",
       " -0.01520447339862585,\n",
       " 0.05478754639625549,\n",
       " 0.013575422577559948,\n",
       " -0.013332493603229523,\n",
       " -0.02370697446167469,\n",
       " -0.050329092890024185,\n",
       " -0.03443870320916176,\n",
       " -0.039154376834630966,\n",
       " 0.005440886598080397,\n",
       " 0.030837643891572952,\n",
       " 0.00736645283177495,\n",
       " 0.013854076154530048,\n",
       " 0.0019666498992592096,\n",
       " -0.014947254210710526,\n",
       " -0.028122559189796448,\n",
       " -0.04652797430753708,\n",
       " -0.021106209605932236,\n",
       " -0.05144370719790459,\n",
       " 0.0006033024983480573,\n",
       " -0.0041262139566242695,\n",
       " -0.026707857847213745,\n",
       " -0.031837936490774155,\n",
       " 0.026507798582315445,\n",
       " -0.022435171529650688,\n",
       " -0.002520384266972542,\n",
       " -0.013425378128886223,\n",
       " 0.0577884279191494,\n",
       " -0.025850461795926094,\n",
       " -0.007173539139330387,\n",
       " -0.04066910594701767,\n",
       " -0.024935906752943993,\n",
       " 0.03160930052399635,\n",
       " -0.0075593669898808,\n",
       " 0.025993362069129944,\n",
       " 0.07253562659025192,\n",
       " 0.04089774563908577,\n",
       " 0.009595680050551891,\n",
       " -0.009359896183013916,\n",
       " -0.016176188364624977,\n",
       " -0.018991300836205482,\n",
       " 0.011010382324457169,\n",
       " -0.05112932622432709,\n",
       " 0.01727651059627533,\n",
       " 0.044155847281217575,\n",
       " 0.024835878983139992,\n",
       " 0.01770520955324173,\n",
       " -0.03635355085134506,\n",
       " 0.01870550401508808,\n",
       " -0.009359896183013916,\n",
       " 0.03598201274871826,\n",
       " -0.0009654625901021063,\n",
       " -0.0050550587475299835,\n",
       " -0.0038296980783343315,\n",
       " 0.01794813759624958,\n",
       " -0.03715378791093826,\n",
       " -0.0030794774647802114,\n",
       " -0.027693862095475197,\n",
       " 0.015490271151065826,\n",
       " -0.04541335999965668,\n",
       " 0.009445635601878166,\n",
       " 0.0019970161374658346,\n",
       " -0.018862692639231682,\n",
       " -0.014247048646211624,\n",
       " -9.23262705327943e-05,\n",
       " -0.0592745803296566,\n",
       " -0.021020470187067986,\n",
       " -0.0283797774463892,\n",
       " -0.013303914107382298,\n",
       " 0.04498466104269028,\n",
       " -0.03966881334781647,\n",
       " 0.06373303383588791,\n",
       " -0.013475392945110798,\n",
       " -0.0004956815391778946,\n",
       " 0.018919851630926132,\n",
       " -0.050900690257549286,\n",
       " -0.056473758071660995,\n",
       " -0.019077040255069733,\n",
       " -0.0030330352019518614,\n",
       " 0.032981131225824356,\n",
       " -0.01700500212609768,\n",
       " -0.008466776460409164,\n",
       " -0.02279241941869259,\n",
       " 0.04312697425484657,\n",
       " -0.032781071960926056,\n",
       " 0.0289227943867445,\n",
       " 0.0010717438999563456,\n",
       " -0.003433152800425887,\n",
       " 0.021334847435355186,\n",
       " 0.02354978583753109,\n",
       " -0.031152021139860153,\n",
       " 0.05238683894276619,\n",
       " -0.04275543615221977,\n",
       " -0.06916320323944092,\n",
       " -0.012139285914599895,\n",
       " -0.03578195720911026,\n",
       " -0.006894885562360287,\n",
       " -0.04684235155582428,\n",
       " 0.04238389804959297,\n",
       " -0.00878115464001894,\n",
       " -0.04246963560581207,\n",
       " -0.03441012278199196,\n",
       " 0.02334972657263279,\n",
       " -0.026893626898527145,\n",
       " -0.031523559242486954,\n",
       " -0.015861809253692627,\n",
       " -0.020706091076135635,\n",
       " 0.015661749988794327,\n",
       " 0.021534906700253487,\n",
       " -0.038268402218818665,\n",
       " 0.02397848293185234,\n",
       " 0.015233052894473076,\n",
       " 0.004779977723956108,\n",
       " 0.06499055027961731,\n",
       " 0.006541210226714611,\n",
       " -0.010188711807131767,\n",
       " 0.013732611201703548,\n",
       " 0.02279241941869259,\n",
       " 0.01200353167951107,\n",
       " 0.025207415223121643,\n",
       " 0.0023917751386761665,\n",
       " -0.02452149987220764,\n",
       " -0.01747656986117363,\n",
       " -0.00860967580229044,\n",
       " 0.00367965386249125,\n",
       " 0.001882696757093072,\n",
       " 0.024907328188419342,\n",
       " 0.015718910843133926,\n",
       " 0.07527928799390793,\n",
       " -0.012146431021392345,\n",
       " 0.01720506139099598,\n",
       " 0.04175513982772827,\n",
       " 0.05384441092610359,\n",
       " 0.025693273171782494,\n",
       " -0.007237843703478575,\n",
       " 0.030551845207810402,\n",
       " -0.08436767756938934,\n",
       " 0.00014155950339045376,\n",
       " -0.05458748713135719,\n",
       " -0.01180347241461277,\n",
       " -0.00879544485360384,\n",
       " -0.01711932197213173,\n",
       " 0.025007357820868492,\n",
       " -0.032266635447740555,\n",
       " -0.055873580276966095,\n",
       " -0.028022529557347298,\n",
       " -0.04089774563908577,\n",
       " -0.015947548672556877,\n",
       " -0.021534906700253487,\n",
       " 0.06739125400781631,\n",
       " 0.021906444802880287,\n",
       " 0.007838020101189613,\n",
       " 0.005530198570340872,\n",
       " 0.027322323992848396,\n",
       " 0.07076367735862732,\n",
       " -0.009509940631687641,\n",
       " 0.004508469253778458,\n",
       " -0.013203884474933147,\n",
       " 0.04152650386095047,\n",
       " -0.03406716510653496,\n",
       " -0.03678224980831146,\n",
       " 0.0043655699118971825,\n",
       " 0.044041529297828674,\n",
       " -0.011510529555380344,\n",
       " 0.044527385383844376,\n",
       " -0.05315849557518959,\n",
       " -0.008738284930586815,\n",
       " -0.030809063464403152,\n",
       " -0.03618207201361656,\n",
       " 0.004519186448305845,\n",
       " -0.009002648293972015,\n",
       " 0.07516497373580933,\n",
       " -0.07099231332540512,\n",
       " -0.016261927783489227,\n",
       " 0.0605892539024353,\n",
       " 0.04684235155582428,\n",
       " -0.006784138735383749,\n",
       " 0.0018308957805857062,\n",
       " -0.014075569808483124,\n",
       " -0.0006930610397830606,\n",
       " 0.010988947004079819,\n",
       " 0.03309544920921326,\n",
       " 0.03741100803017616,\n",
       " 0.01703358255326748,\n",
       " -0.01870550401508808,\n",
       " 0.026522088795900345,\n",
       " 0.016604885458946228,\n",
       " 0.0290799830108881,\n",
       " -0.02940865233540535,\n",
       " 0.06807716935873032,\n",
       " 0.03969739377498627,\n",
       " -0.0145471366122365,\n",
       " 0.008281007409095764,\n",
       " -0.031523559242486954,\n",
       " -0.07893750816583633,\n",
       " 0.031209181994199753,\n",
       " 0.02902282401919365,\n",
       " -0.0011351554421707988,\n",
       " 0.025493213906884193,\n",
       " 0.032981131225824356,\n",
       " 0.01721935160458088,\n",
       " -0.04195519909262657,\n",
       " 0.027665281668305397,\n",
       " -0.0009105357457883656,\n",
       " 0.025350315496325493,\n",
       " 0.004383432678878307,\n",
       " -0.0588173046708107,\n",
       " 0.03366704657673836,\n",
       " -0.009538520127534866,\n",
       " 0.0040761991403996944,\n",
       " 0.010610264725983143,\n",
       " 0.05478754639625549,\n",
       " -0.04558483883738518,\n",
       " -0.008023789152503014,\n",
       " 0.016176188364624977,\n",
       " -0.020949020981788635,\n",
       " -0.08471063524484634,\n",
       " -0.020363133400678635,\n",
       " -0.008902618661522865,\n",
       " 0.0030759049113839865,\n",
       " 0.025621823966503143,\n",
       " -0.007745135575532913,\n",
       " 0.04912874102592468,\n",
       " -0.020620351657271385,\n",
       " -0.006601942237466574,\n",
       " 0.000875704048667103,\n",
       " -0.004387004766613245,\n",
       " 0.033066872507333755,\n",
       " -0.04015466943383217,\n",
       " 0.021963603794574738,\n",
       " -0.006923465523868799,\n",
       " -0.030894802883267403,\n",
       " 0.02427857182919979,\n",
       " -0.037839703261852264,\n",
       " 0.02337830699980259,\n",
       " 0.013060985133051872,\n",
       " -0.014247048646211624,\n",
       " 0.04264111444354057,\n",
       " 0.024750137701630592,\n",
       " 0.007988064549863338,\n",
       " -0.02465010993182659,\n",
       " -0.02276383899152279,\n",
       " 0.019977305084466934,\n",
       " 0.04629933461546898,\n",
       " -0.004201236180961132,\n",
       " -0.044527385383844376,\n",
       " -0.08122389763593674,\n",
       " -0.011296180076897144,\n",
       " 0.06384735554456711,\n",
       " 0.014690035954117775,\n",
       " -0.004815702326595783,\n",
       " -0.01721935160458088,\n",
       " 0.04084058478474617,\n",
       " -0.03549615666270256,\n",
       " 0.031980838626623154,\n",
       " -0.04186946153640747,\n",
       " 0.016261927783489227,\n",
       " -0.04089774563908577,\n",
       " 0.026422059163451195,\n",
       " 0.02424999140202999,\n",
       " 0.08213844895362854,\n",
       " -0.032180897891521454,\n",
       " 0.010360190644860268,\n",
       " -0.0150901535525918,\n",
       " -0.00589816365391016,\n",
       " -0.004930021706968546,\n",
       " 0.003722523804754019,\n",
       " -0.02955155074596405,\n",
       " -0.026393478736281395,\n",
       " 0.010938932187855244,\n",
       " 0.04867146164178848,\n",
       " -0.025421764701604843,\n",
       " -0.028251169249415398,\n",
       " -0.03578195720911026,\n",
       " 0.008123818784952164,\n",
       " 0.032295215874910355,\n",
       " -0.01428991835564375,\n",
       " -0.016119027510285378,\n",
       " -0.03012314811348915,\n",
       " -0.00888832937926054,\n",
       " -0.009581389836966991,\n",
       " -0.013832640834152699,\n",
       " -0.02389274351298809,\n",
       " 0.05341571569442749,\n",
       " 0.022177953273057938,\n",
       " -0.01870550401508808,\n",
       " 0.06310427933931351,\n",
       " -0.05298701673746109,\n",
       " 0.01774807833135128,\n",
       " 0.01200353167951107,\n",
       " 0.008016644045710564,\n",
       " -0.02364981546998024,\n",
       " -0.0064626154489815235,\n",
       " 0.03358130902051926,\n",
       " 0.009938638657331467,\n",
       " 0.019662927836179733,\n",
       " -0.01719077117741108,\n",
       " -0.007302148267626762,\n",
       " -0.01771949790418148,\n",
       " 0.03603917360305786,\n",
       " 0.032638173550367355,\n",
       " -0.00859538558870554,\n",
       " -0.0286084171384573,\n",
       " -0.022035054862499237,\n",
       " -0.00568024255335331,\n",
       " 0.04064052924513817,\n",
       " 0.003334909677505493,\n",
       " -0.028279747813940048,\n",
       " 0.02905140444636345,\n",
       " 0.019148491322994232,\n",
       " 0.028151139616966248,\n",
       " 0.0075879464857280254,\n",
       " 0.0008832955500110984,\n",
       " 0.01483293529599905,\n",
       " -0.06727693229913712,\n",
       " -0.06699113547801971,\n",
       " -0.018948432058095932,\n",
       " 0.0074521922506392,\n",
       " 0.016590595245361328,\n",
       " -0.005398016422986984,\n",
       " -0.04275543615221977,\n",
       " 0.027493802830576897,\n",
       " 0.027422353625297546,\n",
       " -0.0149186747148633,\n",
       " -0.024864457547664642,\n",
       " 0.07139243185520172,\n",
       " 0.006819863338023424,\n",
       " 0.03523894026875496,\n",
       " 0.019319970160722733,\n",
       " 0.0585886649787426,\n",
       " -0.0036617915611714125,\n",
       " 0.006258984096348286,\n",
       " 0.04729963093996048,\n",
       " -0.019577188417315483,\n",
       " 0.005833859089761972,\n",
       " 0.013946960680186749,\n",
       " -0.030894802883267403,\n",
       " 0.005730257369577885,\n",
       " 0.012325054965913296,\n",
       " 0.010310175828635693,\n",
       " 0.020848991349339485,\n",
       " 0.03483882173895836,\n",
       " 0.061903927475214005,\n",
       " -0.02975161001086235,\n",
       " 0.038096923381090164,\n",
       " 0.021692095324397087,\n",
       " 0.009181272238492966,\n",
       " 0.04695667326450348,\n",
       " 0.03501030057668686,\n",
       " -0.0147614860907197,\n",
       " -0.019048461690545082,\n",
       " 0.0574740506708622,\n",
       " -0.044070109724998474,\n",
       " -0.022435171529650688,\n",
       " -0.00018543402256909758,\n",
       " -0.030780484899878502,\n",
       " 0.008488211780786514,\n",
       " 0.01153910905122757,\n",
       " 0.025407474488019943,\n",
       " 0.03495313972234726,\n",
       " -0.031037703156471252,\n",
       " -0.044727444648742676,\n",
       " -0.015847519040107727,\n",
       " -0.01804816722869873,\n",
       " -0.009981508366763592,\n",
       " 0.01730509102344513,\n",
       " 0.05144370719790459,\n",
       " -0.009752869606018066,\n",
       " 0.025250285863876343,\n",
       " 0.026907917112112045,\n",
       " 0.043812889605760574,\n",
       " -0.006066070403903723,\n",
       " 0.037782542407512665,\n",
       " 0.009495650418102741,\n",
       " -0.03641071170568466,\n",
       " -0.010074391961097717,\n",
       " -0.014975834637880325,\n",
       " 0.02352120541036129,\n",
       " -0.016533436253666878,\n",
       " 0.021734965965151787,\n",
       " -0.04535619914531708,\n",
       " -0.0590459443628788,\n",
       " -0.003413504222407937,\n",
       " 0.004540621768683195,\n",
       " -0.04692809283733368,\n",
       " -0.0015245556132867932,\n",
       " 0.02332114614546299,\n",
       " -0.04024041071534157,\n",
       " -0.0008158650016412139,\n",
       " -0.0026972221676260233,\n",
       " -0.01717648096382618,\n",
       " -0.0010520952055230737,\n",
       " -0.02277812920510769,\n",
       " -0.019462868571281433,\n",
       " -0.01841970533132553,\n",
       " 0.01124616526067257,\n",
       " -0.019377129152417183,\n",
       " -0.026150550693273544,\n",
       " 0.02356407605111599,\n",
       " 0.0044227298349142075,\n",
       " 0.04215525835752487,\n",
       " -0.04084058478474617,\n",
       " -0.004872862249612808,\n",
       " 0.020191654562950134,\n",
       " 0.010260161012411118,\n",
       " 0.003038393799215555,\n",
       " 0.013475392945110798,\n",
       " -0.0016978209605440497,\n",
       " -0.004487034399062395,\n",
       " -0.03469592332839966,\n",
       " -0.032923970371484756,\n",
       " 0.032666753977537155,\n",
       " 0.022235112264752388,\n",
       " -0.038811419159173965,\n",
       " -0.01781952753663063,\n",
       " 0.01161770336329937,\n",
       " 0.02250662073493004,\n",
       " 0.08259572833776474,\n",
       " -0.009324171580374241,\n",
       " -0.03358130902051926,\n",
       " 0.0151044437661767,\n",
       " -0.0037046612706035376,\n",
       " -0.0016415544087067246,\n",
       " 0.03452444449067116,\n",
       " 0.01743370108306408,\n",
       " -0.01726222224533558,\n",
       " 0.0149043845012784,\n",
       " 0.0019148490391671658,\n",
       " -0.032781071960926056,\n",
       " -0.0028347624465823174,\n",
       " 0.0286084171384573,\n",
       " -0.05210104212164879,\n",
       " 0.0569881945848465,\n",
       " 0.012346489354968071,\n",
       " -0.010781743563711643,\n",
       " 0.008309587836265564,\n",
       " 0.0578170083463192,\n",
       " 0.016004707664251328,\n",
       " 0.010560249909758568,\n",
       " -0.02292102947831154,\n",
       " 0.013732611201703548,\n",
       " 0.022392302751541138,\n",
       " -0.006802001036703587,\n",
       " -0.02403564192354679,\n",
       " -0.03372420743107796,\n",
       " -0.015447401441633701,\n",
       " -0.012539403513073921,\n",
       " -0.006919892970472574,\n",
       " -0.030951963737607002,\n",
       " 0.015161603689193726,\n",
       " -0.025235995650291443,\n",
       " -0.019963016733527184,\n",
       " -0.00913125742226839,\n",
       " 0.019134201109409332,\n",
       " 0.01501870434731245,\n",
       " -0.037668224424123764,\n",
       " -0.01757659949362278,\n",
       " -0.02427857182919979,\n",
       " -0.009717144072055817,\n",
       " -0.04669945314526558,\n",
       " 0.0004617429804056883,\n",
       " 0.020763251930475235,\n",
       " 0.020563192665576935,\n",
       " -0.032609593123197556,\n",
       " 0.032809652388095856,\n",
       " 0.015318792313337326,\n",
       " -0.020563192665576935,\n",
       " -0.006294709164649248,\n",
       " -0.010996092110872269,\n",
       " 0.04807128384709358,\n",
       " 0.03346698731184006,\n",
       " 0.0041226414032280445,\n",
       " -0.007680831011384726,\n",
       " 0.0077379909344017506,\n",
       " 0.009609970264136791,\n",
       " 0.011603414081037045,\n",
       " -0.04184088110923767,\n",
       " 0.039182957261800766,\n",
       " -0.010581684298813343,\n",
       " 0.01153910905122757,\n",
       " -0.006191106978803873,\n",
       " 0.032380953431129456,\n",
       " 0.003170575713738799,\n",
       " 0.006966335233300924,\n",
       " -0.003054470056667924,\n",
       " -0.010110117495059967,\n",
       " -0.020620351657271385,\n",
       " -0.020063044503331184,\n",
       " 0.02279241941869259,\n",
       " 0.026507798582315445,\n",
       " -0.03924011439085007,\n",
       " 0.005080066155642271,\n",
       " -0.019620057195425034,\n",
       " 0.008723994717001915,\n",
       " 0.02420712076127529,\n",
       " 0.0029865929391235113,\n",
       " -0.012517968192696571,\n",
       " 0.05361577495932579,\n",
       " 0.007080654613673687,\n",
       " 0.0049943262711167336,\n",
       " 0.010931787081062794,\n",
       " -0.025478923693299294,\n",
       " -0.00714138662442565,\n",
       " 0.006902030669152737,\n",
       " -0.019591478630900383,\n",
       " -0.013553987257182598,\n",
       " 0.0032527425792068243,\n",
       " -0.0013664733851328492,\n",
       " -0.013682596385478973,\n",
       " -0.009445635601878166,\n",
       " -0.04847140237689018,\n",
       " 0.019734377041459084,\n",
       " -0.008059513755142689,\n",
       " 0.01690497249364853,\n",
       " 0.010788888670504093,\n",
       " 0.01181061752140522,\n",
       " -0.024607239291071892,\n",
       " 0.04547052085399628,\n",
       " -0.015661749988794327,\n",
       " -0.0022113649174571037,\n",
       " 0.003479595063254237,\n",
       " 0.022063633427023888,\n",
       " 0.03466734290122986,\n",
       " 0.010488799773156643,\n",
       " -0.024821588769555092,\n",
       " 0.02400706335902214,\n",
       " -0.01779094897210598,\n",
       " -0.018734082579612732,\n",
       " -0.0071056620217859745,\n",
       " -0.03535325825214386,\n",
       " -0.007212836295366287,\n",
       " -0.015576010569930077,\n",
       " -0.007269995752722025,\n",
       " -0.026336319744586945,\n",
       " 0.012382213957607746,\n",
       " 0.002590047661215067,\n",
       " 0.0010199429234489799,\n",
       " -0.021963603794574738,\n",
       " 0.0011271173134446144,\n",
       " -0.0022327997721731663,\n",
       " 0.05147228762507439,\n",
       " -0.010281596332788467,\n",
       " -0.019134201109409332,\n",
       " 8.451147004961967e-05,\n",
       " 0.009767158888280392,\n",
       " 0.005523053463548422,\n",
       " 0.02424999140202999,\n",
       " -0.01794813759624958,\n",
       " -0.03978313133120537,\n",
       " 0.01664775423705578,\n",
       " 0.009502795524895191,\n",
       " -0.021706385537981987,\n",
       " -0.025207415223121643,\n",
       " 0.00886689405888319,\n",
       " 0.011181861162185669,\n",
       " 0.021163368597626686,\n",
       " -0.01180347241461277,\n",
       " 0.027122264727950096,\n",
       " 0.032466694712638855,\n",
       " 0.01520447339862585,\n",
       " -0.012803766876459122,\n",
       " 0.004447737243026495,\n",
       " -0.010688859038054943,\n",
       " -0.01826251484453678,\n",
       " 0.03329550847411156,\n",
       " -0.010052957572042942,\n",
       " -0.024850167334079742,\n",
       " -0.014790065586566925,\n",
       " 0.027808181941509247,\n",
       " 0.015804650261998177,\n",
       " 0.022435171529650688,\n",
       " -0.03498172014951706,\n",
       " 0.009445635601878166,\n",
       " 0.005548060871660709,\n",
       " 0.0292943324893713,\n",
       " -0.015561721287667751,\n",
       " 0.044470224529504776,\n",
       " -0.02289244905114174,\n",
       " -0.016804944723844528,\n",
       " 0.021506326273083687,\n",
       " 0.0287227351218462,\n",
       " -0.006962762679904699,\n",
       " 0.026536379009485245,\n",
       " 0.018619762733578682,\n",
       " -0.02330685593187809,\n",
       " -0.01756230928003788,\n",
       " 0.001968436175957322,\n",
       " -0.009838609024882317,\n",
       " -0.009602825157344341,\n",
       " -0.03126634284853935,\n",
       " 0.012903796508908272,\n",
       " -0.04761400818824768,\n",
       " 0.004079771693795919,\n",
       " 0.0019434288842603564,\n",
       " 0.008216703310608864,\n",
       " -0.033238351345062256,\n",
       " 0.003483167616650462,\n",
       " 0.032123737037181854,\n",
       " -0.001638874993659556,\n",
       " -0.0044155847281217575,\n",
       " -0.011317615397274494,\n",
       " 0.021977894008159637,\n",
       " 0.003343840828165412,\n",
       " 0.06853444874286652,\n",
       " 0.025678982958197594,\n",
       " 0.007259278558194637,\n",
       " 0.00841676164418459,\n",
       " 0.010910352692008018,\n",
       " -0.031923677772283554,\n",
       " -0.04044046998023987,\n",
       " 0.010781743563711643,\n",
       " 0.016604885458946228,\n",
       " -0.003601059317588806,\n",
       " 0.04126928374171257,\n",
       " -0.004819274879992008,\n",
       " 0.0037475309800356627,\n",
       " -0.011660573072731495,\n",
       " 0.03046610578894615,\n",
       " 0.013775480911135674,\n",
       " -0.0074521922506392,\n",
       " -0.019720086827874184,\n",
       " -0.010124406777322292,\n",
       " -0.038382720202207565,\n",
       " -0.01173916831612587,\n",
       " 0.013153869658708572,\n",
       " -0.021720675751566887,\n",
       " 0.02960871160030365,\n",
       " 0.0028758461121469736,\n",
       " 0.021349137648940086,\n",
       " -0.012546548619866371,\n",
       " 0.0151044437661767,\n",
       " 0.01779094897210598,\n",
       " 0.005987475626170635,\n",
       " 0.0012235742760822177,\n",
       " 0.016476275399327278,\n",
       " -0.0031509269028902054,\n",
       " 0.010381625965237617,\n",
       " -0.03423864394426346,\n",
       " 0.01761947013437748,\n",
       " 0.022649521008133888,\n",
       " 0.055730681866407394,\n",
       " 0.02436431124806404,\n",
       " 0.0014441749081015587,\n",
       " 0.0015906465705484152,\n",
       " 0.010988947004079819,\n",
       " -0.007773715537041426,\n",
       " 0.02426428161561489,\n",
       " 0.0020327409729361534,\n",
       " 0.01453999150544405,\n",
       " -0.012382213957607746,\n",
       " -0.030637584626674652,\n",
       " -0.007212836295366287,\n",
       " -0.007062792312353849,\n",
       " -0.010274451225996017,\n",
       " -0.03392426669597626,\n",
       " -0.03601059317588806,\n",
       " 0.0037118063773959875,\n",
       " -0.038611359894275665,\n",
       " 0.007623671554028988,\n",
       " 0.0033456271048635244,\n",
       " -0.000569363939575851,\n",
       " 0.001685317256487906,\n",
       " 0.0003927494690287858,\n",
       " 0.006530492566525936,\n",
       " 0.032752491533756256,\n",
       " -0.01483293529599905,\n",
       " 0.04158366098999977,\n",
       " 0.007673686370253563,\n",
       " -0.02456437051296234,\n",
       " -0.020777542144060135,\n",
       " -0.004608498886227608,\n",
       " -0.018848402425646782,\n",
       " -0.056245118379592896,\n",
       " -0.024978777393698692,\n",
       " 0.01463287603110075,\n",
       " 0.025536084547638893,\n",
       " -0.032723914831876755,\n",
       " 0.02918001264333725,\n",
       " 0.00603034533560276,\n",
       " -0.02992308884859085,\n",
       " -0.0014129156479611993,\n",
       " -0.013639726676046848,\n",
       " -0.0289799552410841,\n",
       " 0.012467953376471996,\n",
       " 0.009459925815463066,\n",
       " -0.03032320737838745,\n",
       " -0.025350315496325493,\n",
       " -0.0034117179457098246,\n",
       " 0.003370634512975812,\n",
       " -0.0289799552410841,\n",
       " 0.01733367145061493,\n",
       " 0.025493213906884193,\n",
       " 0.016133317723870277,\n",
       " 0.020577482879161835,\n",
       " 0.02262094058096409,\n",
       " 0.022435171529650688,\n",
       " -0.010060102678835392,\n",
       " 0.005262262653559446,\n",
       " 0.04761400818824768,\n",
       " 0.02975161001086235,\n",
       " -0.019534317776560783,\n",
       " 0.027550961822271347,\n",
       " -0.010860337875783443,\n",
       " -0.019862987101078033,\n",
       " 0.005901736207306385,\n",
       " -0.00933131668716669,\n",
       " 0.045299042016267776,\n",
       " -0.02259236015379429,\n",
       " -0.025936201214790344,\n",
       " -0.005790989380329847,\n",
       " 0.006334006320685148,\n",
       " 0.02938007190823555,\n",
       " 0.007802295498549938,\n",
       " -0.004597781226038933,\n",
       " -0.025164546445012093,\n",
       " -0.02995166927576065,\n",
       " 0.0299802478402853,\n",
       " -0.01438280288130045,\n",
       " 0.01681923307478428,\n",
       " 0.009860043413937092,\n",
       " 0.02270667999982834,\n",
       " -0.01444710697978735,\n",
       " -0.012575128115713596,\n",
       " -0.021163368597626686,\n",
       " 0.0144685423001647,\n",
       " -0.02403564192354679,\n",
       " 0.002050603274255991,\n",
       " 0.02324969694018364,\n",
       " 0.02905140444636345,\n",
       " -0.0580742284655571,\n",
       " 0.008166688494384289,\n",
       " 0.03672508895397186,\n",
       " -0.001478113466873765,\n",
       " 0.007005632389336824,\n",
       " -1.622966374270618e-05,\n",
       " 0.04261253774166107,\n",
       " 0.02922288328409195,\n",
       " 0.02423570118844509,\n",
       " -0.0070592197589576244,\n",
       " 0.04484176263213158,\n",
       " 0.00581242423504591,\n",
       " -0.025536084547638893,\n",
       " 0.011131846345961094,\n",
       " 0.05115790665149689,\n",
       " -0.007019922602921724,\n",
       " 0.01804816722869873,\n",
       " 0.019491448998451233,\n",
       " -0.0603606142103672,\n",
       " -0.02392132394015789,\n",
       " -0.01677636429667473,\n",
       " 0.05450174957513809,\n",
       " 0.015747489407658577,\n",
       " -0.0151187339797616,\n",
       " -0.0050586313009262085,\n",
       " 0.004922877065837383,\n",
       " -0.014061279594898224,\n",
       " 0.031494978815317154,\n",
       " 0.007788005284965038,\n",
       " -0.001197673729620874,\n",
       " 0.06653385609388351,\n",
       " -0.02316395752131939,\n",
       " 0.013661161996424198,\n",
       " 0.014718615449965,\n",
       " 0.0009958286536857486,\n",
       " -0.03986887261271477,\n",
       " -0.01694784313440323,\n",
       " -0.0010583470575511456,\n",
       " 0.0581313893198967,\n",
       " -0.01771949790418148,\n",
       " 0.012124995701014996,\n",
       " -0.02364981546998024,\n",
       " -0.018805531784892082,\n",
       " 9.199135092785582e-05,\n",
       " -0.004254823084920645,\n",
       " -0.002254234626889229,\n",
       " 0.027979660779237747,\n",
       " -0.006794856395572424,\n",
       " 0.020220234990119934,\n",
       " -0.010310175828635693,\n",
       " 0.0023453328758478165,\n",
       " -0.0011565902968868613,\n",
       " -0.007077082060277462,\n",
       " -0.021577777341008186,\n",
       " 0.0019148490391671658,\n",
       " 0.02955155074596405,\n",
       " -0.031552139669656754,\n",
       " 0.005019333679229021,\n",
       " -0.014804355800151825,\n",
       " 0.0143613675609231,\n",
       " -0.010788888670504093,\n",
       " -0.0143613675609231,\n",
       " -0.0040690540336072445,\n",
       " -0.006712689064443111,\n",
       " -0.008673980832099915,\n",
       " -0.020848991349339485,\n",
       " -0.044927503913640976,\n",
       " 0.025879042223095894,\n",
       " 0.012775187380611897,\n",
       " -0.03140924125909805,\n",
       " -0.008395327255129814,\n",
       " 0.04024041071534157,\n",
       " 0.0014477473450824618,\n",
       " 0.020577482879161835,\n",
       " 0.0298659298568964,\n",
       " -0.03661077097058296,\n",
       " 0.012746606953442097,\n",
       " 0.01826251484453678,\n",
       " 0.021720675751566887,\n",
       " 0.009795739315450191,\n",
       " 0.03924011439085007,\n",
       " 0.031180601567029953,\n",
       " 0.012860926799476147,\n",
       " 0.011024671606719494,\n",
       " -0.016447696834802628,\n",
       " 0.01749086007475853,\n",
       " -0.0061196573078632355,\n",
       " -0.02406422235071659,\n",
       " -0.028394067659974098,\n",
       " 0.021734965965151787,\n",
       " 0.0605892539024353,\n",
       " 0.02269238978624344,\n",
       " 0.006977052427828312,\n",
       " -0.01179632730782032,\n",
       " -0.005798134487122297,\n",
       " -0.025078807026147842,\n",
       " 0.012325054965913296,\n",
       " 0.012696592137217522,\n",
       " 0.0073950327932834625,\n",
       " -0.02407851256430149,\n",
       " 0.015918968245387077,\n",
       " 0.022663811221718788,\n",
       " -0.006323288660496473,\n",
       " -0.020806120708584785,\n",
       " -0.056873876601457596,\n",
       " 0.0008364067762158811,\n",
       " 0.031094862148165703,\n",
       " -0.0018380407709628344,\n",
       " 0.008259573020040989,\n",
       " -0.025207415223121643,\n",
       " 0.031837936490774155,\n",
       " -0.03612491488456726,\n",
       " 0.010746018961071968,\n",
       " 0.006051780190318823,\n",
       " -0.006651957053691149,\n",
       " 0.01143193431198597,\n",
       " 0.02240659110248089,\n",
       " 0.0005809745052829385,\n",
       " 0.007516496814787388,\n",
       " -0.03398142755031586,\n",
       " -0.003265246283262968,\n",
       " 0.010638844221830368,\n",
       " 0.01767662912607193,\n",
       " -0.025550374761223793,\n",
       " -0.015833228826522827,\n",
       " 0.014675745740532875,\n",
       " -0.012103560380637646,\n",
       " 0.0007354842382483184,\n",
       " 0.03538183867931366,\n",
       " -0.013932670466601849,\n",
       " -0.030951963737607002,\n",
       " -0.010260161012411118,\n",
       " -0.0144828325137496,\n",
       " 0.027522383257746696,\n",
       " -0.006680537015199661,\n",
       " -0.019963016733527184,\n",
       " -0.021063338965177536,\n",
       " 0.009367041289806366,\n",
       " -0.05524482578039169,\n",
       " 0.006680537015199661,\n",
       " 0.006316144019365311,\n",
       " -0.0014629303477704525,\n",
       " 0.01727651059627533,\n",
       " -0.019119910895824432,\n",
       " -0.0043619973585009575,\n",
       " -0.022292273119091988,\n",
       " -0.015947548672556877,\n",
       " 0.025221705436706543,\n",
       " -0.01191064715385437,\n",
       " 0.026664987206459045,\n",
       " 0.005394444335252047,\n",
       " 0.03383852541446686,\n",
       " -0.015876099467277527,\n",
       " 0.0147471958771348,\n",
       " -0.024850167334079742,\n",
       " 0.009595680050551891,\n",
       " 0.010860337875783443,\n",
       " 0.02339259721338749,\n",
       " -0.019105620682239532,\n",
       " 0.04807128384709358,\n",
       " 0.032180897891521454,\n",
       " 0.009838609024882317,\n",
       " -0.03372420743107796,\n",
       " -0.028051109984517097,\n",
       " 0.03463876247406006,\n",
       " -0.010145842097699642,\n",
       " 0.013189594261348248,\n",
       " 0.015661749988794327,\n",
       " 0.00018308957805857062,\n",
       " 0.009702854789793491,\n",
       " -0.004883579444140196,\n",
       " 0.012303619645535946,\n",
       " -0.016204766929149628,\n",
       " -0.005662380252033472,\n",
       " 0.0142684830352664,\n",
       " -0.0029151432681828737,\n",
       " 0.04661371186375618,\n",
       " -0.01215357519686222,\n",
       " 0.008766865357756615,\n",
       " 0.04555625841021538,\n",
       " -0.01700500212609768,\n",
       " -0.007902325130999088,\n",
       " -0.03475308045744896,\n",
       " 0.04584205895662308,\n",
       " 0.019419999793171883,\n",
       " -0.01483293529599905,\n",
       " 0.00037511036498472095,\n",
       " 0.025135966017842293,\n",
       " 0.004111924208700657,\n",
       " 0.027522383257746696,\n",
       " 0.01856260374188423,\n",
       " 0.008009498938918114,\n",
       " 0.013789771124720573,\n",
       " -0.020305974408984184,\n",
       " 0.005080066155642271,\n",
       " -0.019591478630900383,\n",
       " -0.0022345860488712788,\n",
       " 0.043841470032930374,\n",
       " -0.0033188334200531244,\n",
       " -0.022378012537956238,\n",
       " 0.007659396156668663,\n",
       " 0.014790065586566925,\n",
       " 0.019762957468628883,\n",
       " ...]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# example\n",
    "doc_pipeline.client().embeddings.create(input=\"this is some text that will change into a vector\", model=\"text-embedding-3-small\").data[0].embedding"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2975f4f9",
   "metadata": {},
   "source": [
    "# 5. Store results\n",
    "Where you store the result of the pipeline is heavily context and situation dependent.\n",
    "\n",
    "Here for simplicity we put it into a pandas dataframe to easily display. One could then create a parquet, or CSV, from this dataframe, and then bulk ingest into a vector store or database. Or you could just directly write to them from inside this function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "b9a1e2e8",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-03T22:36:42.677105Z",
     "start_time": "2024-05-03T22:36:42.414916Z"
    },
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 10.0.1 (20240210.2158)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"871pt\" height=\"451pt\"\n",
       " viewBox=\"0.00 0.00 870.50 450.80\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 446.8)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-446.8 866.5,-446.8 866.5,4 -4,4\"/>\n",
       "<g id=\"clust1\" class=\"cluster\">\n",
       "<title>cluster__legend</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" points=\"25.88,-304.8 25.88,-434.8 110.72,-434.8 110.72,-304.8 25.88,-304.8\"/>\n",
       "<text text-anchor=\"middle\" x=\"68.3\" y=\"-417.5\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Legend</text>\n",
       "</g>\n",
       "<!-- chunked_document -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>chunked_document</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M540.05,-145.6C540.05,-145.6 414.2,-145.6 414.2,-145.6 408.2,-145.6 402.2,-139.6 402.2,-133.6 402.2,-133.6 402.2,-94 402.2,-94 402.2,-88 408.2,-82 414.2,-82 414.2,-82 540.05,-82 540.05,-82 546.05,-82 552.05,-88 552.05,-94 552.05,-94 552.05,-133.6 552.05,-133.6 552.05,-139.6 546.05,-145.6 540.05,-145.6\"/>\n",
       "<text text-anchor=\"start\" x=\"413\" y=\"-122.5\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">chunked_document</text>\n",
       "<text text-anchor=\"start\" x=\"446\" y=\"-94.5\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">Document</text>\n",
       "</g>\n",
       "<!-- embedded_document -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>embedded_document</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M731.65,-186.6C731.65,-186.6 593.05,-186.6 593.05,-186.6 587.05,-186.6 581.05,-180.6 581.05,-174.6 581.05,-174.6 581.05,-135 581.05,-135 581.05,-129 587.05,-123 593.05,-123 593.05,-123 731.65,-123 731.65,-123 737.65,-123 743.65,-129 743.65,-135 743.65,-135 743.65,-174.6 743.65,-174.6 743.65,-180.6 737.65,-186.6 731.65,-186.6\"/>\n",
       "<text text-anchor=\"start\" x=\"591.85\" y=\"-163.5\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">embedded_document</text>\n",
       "<text text-anchor=\"start\" x=\"631.23\" y=\"-135.5\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">Document</text>\n",
       "</g>\n",
       "<!-- chunked_document&#45;&gt;embedded_document -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>chunked_document&#45;&gt;embedded_document</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M552.39,-130.42C558.07,-131.69 563.84,-132.98 569.62,-134.27\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"568.7,-137.65 579.22,-136.42 570.23,-130.82 568.7,-137.65\"/>\n",
       "</g>\n",
       "<!-- store -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>store</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M850.5,-186.6C850.5,-186.6 784.65,-186.6 784.65,-186.6 778.65,-186.6 772.65,-180.6 772.65,-174.6 772.65,-174.6 772.65,-135 772.65,-135 772.65,-129 778.65,-123 784.65,-123 784.65,-123 850.5,-123 850.5,-123 856.5,-123 862.5,-129 862.5,-135 862.5,-135 862.5,-174.6 862.5,-174.6 862.5,-180.6 856.5,-186.6 850.5,-186.6\"/>\n",
       "<text text-anchor=\"start\" x=\"801.08\" y=\"-163.5\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">store</text>\n",
       "<text text-anchor=\"start\" x=\"783.45\" y=\"-135.5\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">DataFrame</text>\n",
       "</g>\n",
       "<!-- embedded_document&#45;&gt;store -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>embedded_document&#45;&gt;store</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M744,-154.8C749.82,-154.8 755.61,-154.8 761.24,-154.8\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"760.99,-158.3 770.99,-154.8 760.99,-151.3 760.99,-158.3\"/>\n",
       "</g>\n",
       "<!-- html_regex -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>html_regex</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M103.47,-294.6C103.47,-294.6 33.12,-294.6 33.12,-294.6 27.12,-294.6 21.12,-288.6 21.12,-282.6 21.12,-282.6 21.12,-243 21.12,-243 21.12,-237 27.12,-231 33.12,-231 33.12,-231 103.47,-231 103.47,-231 109.47,-231 115.47,-237 115.47,-243 115.47,-243 115.47,-282.6 115.47,-282.6 115.47,-288.6 109.47,-294.6 103.47,-294.6\"/>\n",
       "<text text-anchor=\"start\" x=\"31.92\" y=\"-271.5\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">html_regex</text>\n",
       "<text text-anchor=\"start\" x=\"60.8\" y=\"-243.5\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- raw_document -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>raw_document</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M316.57,-227.6C316.57,-227.6 222.22,-227.6 222.22,-227.6 216.22,-227.6 210.22,-221.6 210.22,-215.6 210.22,-215.6 210.22,-176 210.22,-176 210.22,-170 216.22,-164 222.22,-164 222.22,-164 316.57,-164 316.57,-164 322.57,-164 328.57,-170 328.57,-176 328.57,-176 328.57,-215.6 328.57,-215.6 328.57,-221.6 322.57,-227.6 316.57,-227.6\"/>\n",
       "<text text-anchor=\"start\" x=\"221.02\" y=\"-204.5\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">raw_document</text>\n",
       "<text text-anchor=\"start\" x=\"238.27\" y=\"-176.5\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">Document</text>\n",
       "</g>\n",
       "<!-- html_regex&#45;&gt;raw_document -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>html_regex&#45;&gt;raw_document</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M115.73,-247.18C140.48,-238.85 171.43,-228.43 198.98,-219.16\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"199.95,-222.53 208.31,-216.02 197.72,-215.89 199.95,-222.53\"/>\n",
       "</g>\n",
       "<!-- raw_document&#45;&gt;chunked_document -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>raw_document&#45;&gt;chunked_document</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M328.8,-172.52C348.21,-164.78 370.24,-156.01 391.23,-147.64\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"392.37,-150.95 400.36,-144 389.78,-144.45 392.37,-150.95\"/>\n",
       "</g>\n",
       "<!-- client -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>client</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M499.18,-227.6C499.18,-227.6 455.07,-227.6 455.07,-227.6 449.07,-227.6 443.07,-221.6 443.07,-215.6 443.07,-215.6 443.07,-176 443.07,-176 443.07,-170 449.07,-164 455.07,-164 455.07,-164 499.18,-164 499.18,-164 505.18,-164 511.18,-170 511.18,-176 511.18,-176 511.18,-215.6 511.18,-215.6 511.18,-221.6 505.18,-227.6 499.18,-227.6\"/>\n",
       "<text text-anchor=\"start\" x=\"459.5\" y=\"-204.5\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">client</text>\n",
       "<text text-anchor=\"start\" x=\"453.88\" y=\"-176.5\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">OpenAI</text>\n",
       "</g>\n",
       "<!-- client&#45;&gt;embedded_document -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>client&#45;&gt;embedded_document</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M511.31,-188.38C527.83,-184.68 548.7,-180.01 569.58,-175.33\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"570.35,-178.75 579.34,-173.15 568.82,-171.92 570.35,-178.75\"/>\n",
       "</g>\n",
       "<!-- text_chunker -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>text_chunker</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M361.2,-145.6C361.2,-145.6 177.6,-145.6 177.6,-145.6 171.6,-145.6 165.6,-139.6 165.6,-133.6 165.6,-133.6 165.6,-94 165.6,-94 165.6,-88 171.6,-82 177.6,-82 177.6,-82 361.2,-82 361.2,-82 367.2,-82 373.2,-88 373.2,-94 373.2,-94 373.2,-133.6 373.2,-133.6 373.2,-139.6 367.2,-145.6 361.2,-145.6\"/>\n",
       "<text text-anchor=\"start\" x=\"227.4\" y=\"-122.5\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">text_chunker</text>\n",
       "<text text-anchor=\"start\" x=\"176.4\" y=\"-94.5\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">RecursiveCharacterTextSplitter</text>\n",
       "</g>\n",
       "<!-- text_chunker&#45;&gt;chunked_document -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>text_chunker&#45;&gt;chunked_document</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M373.56,-113.8C379.22,-113.8 384.87,-113.8 390.46,-113.8\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"390.29,-117.3 400.29,-113.8 390.29,-110.3 390.29,-117.3\"/>\n",
       "</g>\n",
       "<!-- html_chunker -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>html_chunker</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M342.45,-63.6C342.45,-63.6 196.35,-63.6 196.35,-63.6 190.35,-63.6 184.35,-57.6 184.35,-51.6 184.35,-51.6 184.35,-12 184.35,-12 184.35,-6 190.35,0 196.35,0 196.35,0 342.45,0 342.45,0 348.45,0 354.45,-6 354.45,-12 354.45,-12 354.45,-51.6 354.45,-51.6 354.45,-57.6 348.45,-63.6 342.45,-63.6\"/>\n",
       "<text text-anchor=\"start\" x=\"224.77\" y=\"-40.5\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">html_chunker</text>\n",
       "<text text-anchor=\"start\" x=\"195.15\" y=\"-12.5\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">HTMLHeaderTextSplitter</text>\n",
       "</g>\n",
       "<!-- html_chunker&#45;&gt;chunked_document -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>html_chunker&#45;&gt;chunked_document</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M351.41,-64.09C364.54,-69.32 378.17,-74.76 391.39,-80.03\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"389.89,-83.2 400.48,-83.65 392.48,-76.69 389.89,-83.2\"/>\n",
       "</g>\n",
       "<!-- _raw_document_inputs -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>_raw_document_inputs</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"101.1,-213.1 35.5,-213.1 35.5,-168.5 101.1,-168.5 101.1,-213.1\"/>\n",
       "<text text-anchor=\"start\" x=\"50.3\" y=\"-185\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">url</text>\n",
       "<text text-anchor=\"start\" x=\"71.3\" y=\"-185\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- _raw_document_inputs&#45;&gt;raw_document -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>_raw_document_inputs&#45;&gt;raw_document</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M101.29,-191.6C127.44,-192.26 165.38,-193.21 198.5,-194.04\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"198.39,-197.54 208.47,-194.3 198.57,-190.55 198.39,-197.54\"/>\n",
       "</g>\n",
       "<!-- _text_chunker_inputs -->\n",
       "<g id=\"node10\" class=\"node\">\n",
       "<title>_text_chunker_inputs</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"136.6,-146.6 0,-146.6 0,-81 136.6,-81 136.6,-146.6\"/>\n",
       "<text text-anchor=\"start\" x=\"24.55\" y=\"-118.5\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">chunk_size</text>\n",
       "<text text-anchor=\"start\" x=\"107.67\" y=\"-118.5\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">int</text>\n",
       "<text text-anchor=\"start\" x=\"14.42\" y=\"-97.5\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">chunk_overlap</text>\n",
       "<text text-anchor=\"start\" x=\"107.67\" y=\"-97.5\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">int</text>\n",
       "</g>\n",
       "<!-- _text_chunker_inputs&#45;&gt;text_chunker -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>_text_chunker_inputs&#45;&gt;text_chunker</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M136.98,-113.8C142.55,-113.8 148.26,-113.8 154.04,-113.8\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"153.71,-117.3 163.71,-113.8 153.71,-110.3 153.71,-117.3\"/>\n",
       "</g>\n",
       "<!-- input -->\n",
       "<g id=\"node11\" class=\"node\">\n",
       "<title>input</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"95.3,-349.1 41.3,-349.1 41.3,-312.5 95.3,-312.5 95.3,-349.1\"/>\n",
       "<text text-anchor=\"middle\" x=\"68.3\" y=\"-325\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">input</text>\n",
       "</g>\n",
       "<!-- function -->\n",
       "<g id=\"node12\" class=\"node\">\n",
       "<title>function</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M90.72,-404.1C90.72,-404.1 45.87,-404.1 45.87,-404.1 39.87,-404.1 33.87,-398.1 33.87,-392.1 33.87,-392.1 33.87,-379.5 33.87,-379.5 33.87,-373.5 39.87,-367.5 45.87,-367.5 45.87,-367.5 90.72,-367.5 90.72,-367.5 96.72,-367.5 102.72,-373.5 102.72,-379.5 102.72,-379.5 102.72,-392.1 102.72,-392.1 102.72,-398.1 96.72,-404.1 90.72,-404.1\"/>\n",
       "<text text-anchor=\"middle\" x=\"68.3\" y=\"-380\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">function</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x1519a29d0>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%incr_cell_to_module doc_pipeline 5 --display\n",
    "import pandas as pd\n",
    "\n",
    "def store(\n",
    "    embedded_document: Document,\n",
    ") -> pd.DataFrame:\n",
    "    \"\"\"Function to index & store the document. Here we just put it into a pandas dataframe.\"\"\"\n",
    "    # make a pandas dataframe from the document\n",
    "    # want a column for the document id, the chunk index, the text, the embedding, and the metadata\n",
    "    rows = []\n",
    "    for chunk in embedded_document.chunks:\n",
    "        # need to \n",
    "        rows.append(\n",
    "            {\n",
    "                \"document_id\": embedded_document.id,\n",
    "                \"chunk_index\": chunk.index,\n",
    "                \"url\": embedded_document.url,\n",
    "                \"text\": chunk.text,\n",
    "                \"embedding\": chunk.embedding,\n",
    "                \"metadata\": chunk.metadata,\n",
    "            }\n",
    "        )\n",
    "    data_set = pd.DataFrame(rows)\n",
    "    # we create an index using the document_id and chunk_index\n",
    "    data_set.set_index([\"document_id\", \"chunk_index\"], inplace=True)\n",
    "    return data_set"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "691b0d11",
   "metadata": {},
   "source": [
    "# Let's run the code!\n",
    "Here we use Hamilton to run the pipeline from start to end, i.e. the picture you see above.\n",
    "\n",
    "To do that we just need to:\n",
    "\n",
    "1. Create a \"driver\"\n",
    "2. Execute the driver providing inputs required. Note we could request any intermediate results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "1574cbe8",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-04T00:41:30.970700Z",
     "start_time": "2024-05-04T00:41:30.362619Z"
    },
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>url</th>\n",
       "      <th>text</th>\n",
       "      <th>embedding</th>\n",
       "      <th>metadata</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>document_id</th>\n",
       "      <th>chunk_index</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"22\" valign=\"top\">b5d59eed-cab7-4ed4-909f-940b4f22c22c</th>\n",
       "      <th>0</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Hamilton is a general-purpose framework to wri...</td>\n",
       "      <td>[-0.02026444487273693, 0.009682427160441875, 0...</td>\n",
       "      <td>{'Header 1': 'Welcome to Hamilton¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Directed Acyclic Graph (DAG) that can be execu...</td>\n",
       "      <td>[-0.03777306526899338, 0.0073875077068805695, ...</td>\n",
       "      <td>{'Header 1': 'Welcome to Hamilton¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>The ABC of Hamilton¶</td>\n",
       "      <td>[-0.01697148010134697, 0.02586626075208187, 0....</td>\n",
       "      <td>{'Header 1': 'Welcome to Hamilton¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Facilitate collaboration. By focusing on funct...</td>\n",
       "      <td>[-0.018855366855859756, -0.00249655288644135, ...</td>\n",
       "      <td>{'Header 1': 'Why should you use Hamilton?¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Visualizations can be generated directly from ...</td>\n",
       "      <td>[-0.047300878912210464, 0.031106166541576385, ...</td>\n",
       "      <td>{'Header 1': 'Why should you use Hamilton?¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Reduce development time. Hamilton dataflows ar...</td>\n",
       "      <td>[-0.013192436657845974, -0.025940468534827232,...</td>\n",
       "      <td>{'Header 1': 'Why should you use Hamilton?¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Hamilton Hub.</td>\n",
       "      <td>[-0.064208984375, -0.01950087398290634, 0.0302...</td>\n",
       "      <td>{'Header 1': 'Why should you use Hamilton?¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Own your platform. Hamilton helps you integrat...</td>\n",
       "      <td>[-0.01695844531059265, -0.023515893146395683, ...</td>\n",
       "      <td>{'Header 1': 'Why should you use Hamilton?¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Scale your dataflow. Hamilton separates transf...</td>\n",
       "      <td>[0.0007336082635447383, 0.012326962314546108, ...</td>\n",
       "      <td>{'Header 1': 'Why should you use Hamilton?¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>enterprise data workloads.</td>\n",
       "      <td>[0.007231128867715597, 0.018477486446499825, 0...</td>\n",
       "      <td>{'Header 1': 'Why should you use Hamilton?¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Here’s a quick overview of benefits that Hamil...</td>\n",
       "      <td>[-0.04146569222211838, 0.01626516319811344, 0....</td>\n",
       "      <td>{'Header 1': 'Why should you use Hamilton?¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Can visualize lineage easily  \\n✅  \\n❌  \\n❌  \\...</td>\n",
       "      <td>[-0.03983769938349724, 0.01923440210521221, 0....</td>\n",
       "      <td>{'Header 1': 'Why should you use Hamilton?¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Runs anywhere python runs  \\n✅  \\n❌  \\n❌  \\n❌ ...</td>\n",
       "      <td>[-0.04583346098661423, 0.022130435332655907, 0...</td>\n",
       "      <td>{'Header 1': 'Why should you use Hamilton?¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>The following diagram gives a simplified overv...</td>\n",
       "      <td>[-0.04830301180481911, 0.021801836788654327, 0...</td>\n",
       "      <td>{'Header 1': 'Architecture Overview¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Functions &amp; Module. Transformations are regula...</td>\n",
       "      <td>[-0.041252683848142624, 0.04047299176454544, 0...</td>\n",
       "      <td>{'Header 1': 'Architecture Overview¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Driver &amp; FunctionGraph. The Driver will automa...</td>\n",
       "      <td>[-0.012794064357876778, 0.018729455769062042, ...</td>\n",
       "      <td>{'Header 1': 'Architecture Overview¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Visualization. The FunctionGraph can be visual...</td>\n",
       "      <td>[-0.045535918325185776, 0.034158188849687576, ...</td>\n",
       "      <td>{'Header 1': 'Architecture Overview¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Execution. When requesting variables, the Driv...</td>\n",
       "      <td>[0.0031941100023686886, 0.030165577307343483, ...</td>\n",
       "      <td>{'Header 1': 'Architecture Overview¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Multiple companies are doing cool stuff with H...</td>\n",
       "      <td>[-0.00601744232699275, 0.008543377742171288, 0...</td>\n",
       "      <td>{'Header 1': 'Who is using Hamilton?¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Transfix - Online featurization and prediction...</td>\n",
       "      <td>[-0.0015230675926432014, 0.010722102597355843,...</td>\n",
       "      <td>{'Header 1': 'Who is using Hamilton?¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Habitat - Time-series feature engineering  \\nU...</td>\n",
       "      <td>[0.009830841794610023, 0.004250616300851107, 0...</td>\n",
       "      <td>{'Header 1': 'Who is using Hamilton?¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>KI - Feature engineering</td>\n",
       "      <td>[-0.010076257400214672, 0.01848677173256874, 0...</td>\n",
       "      <td>{'Header 1': 'Who is using Hamilton?¶'}</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                      url  \\\n",
       "document_id                          chunk_index                                            \n",
       "b5d59eed-cab7-4ed4-909f-940b4f22c22c 0            https://hamilton.dagworks.io/en/latest/   \n",
       "                                     1            https://hamilton.dagworks.io/en/latest/   \n",
       "                                     2            https://hamilton.dagworks.io/en/latest/   \n",
       "                                     3            https://hamilton.dagworks.io/en/latest/   \n",
       "                                     4            https://hamilton.dagworks.io/en/latest/   \n",
       "                                     5            https://hamilton.dagworks.io/en/latest/   \n",
       "                                     6            https://hamilton.dagworks.io/en/latest/   \n",
       "                                     7            https://hamilton.dagworks.io/en/latest/   \n",
       "                                     8            https://hamilton.dagworks.io/en/latest/   \n",
       "                                     9            https://hamilton.dagworks.io/en/latest/   \n",
       "                                     10           https://hamilton.dagworks.io/en/latest/   \n",
       "                                     11           https://hamilton.dagworks.io/en/latest/   \n",
       "                                     12           https://hamilton.dagworks.io/en/latest/   \n",
       "                                     13           https://hamilton.dagworks.io/en/latest/   \n",
       "                                     14           https://hamilton.dagworks.io/en/latest/   \n",
       "                                     15           https://hamilton.dagworks.io/en/latest/   \n",
       "                                     16           https://hamilton.dagworks.io/en/latest/   \n",
       "                                     17           https://hamilton.dagworks.io/en/latest/   \n",
       "                                     18           https://hamilton.dagworks.io/en/latest/   \n",
       "                                     19           https://hamilton.dagworks.io/en/latest/   \n",
       "                                     20           https://hamilton.dagworks.io/en/latest/   \n",
       "                                     21           https://hamilton.dagworks.io/en/latest/   \n",
       "\n",
       "                                                                                               text  \\\n",
       "document_id                          chunk_index                                                      \n",
       "b5d59eed-cab7-4ed4-909f-940b4f22c22c 0            Hamilton is a general-purpose framework to wri...   \n",
       "                                     1            Directed Acyclic Graph (DAG) that can be execu...   \n",
       "                                     2                                         The ABC of Hamilton¶   \n",
       "                                     3            Facilitate collaboration. By focusing on funct...   \n",
       "                                     4            Visualizations can be generated directly from ...   \n",
       "                                     5            Reduce development time. Hamilton dataflows ar...   \n",
       "                                     6                                                Hamilton Hub.   \n",
       "                                     7            Own your platform. Hamilton helps you integrat...   \n",
       "                                     8            Scale your dataflow. Hamilton separates transf...   \n",
       "                                     9                                   enterprise data workloads.   \n",
       "                                     10           Here’s a quick overview of benefits that Hamil...   \n",
       "                                     11           Can visualize lineage easily  \\n✅  \\n❌  \\n❌  \\...   \n",
       "                                     12           Runs anywhere python runs  \\n✅  \\n❌  \\n❌  \\n❌ ...   \n",
       "                                     13           The following diagram gives a simplified overv...   \n",
       "                                     14           Functions & Module. Transformations are regula...   \n",
       "                                     15           Driver & FunctionGraph. The Driver will automa...   \n",
       "                                     16           Visualization. The FunctionGraph can be visual...   \n",
       "                                     17           Execution. When requesting variables, the Driv...   \n",
       "                                     18           Multiple companies are doing cool stuff with H...   \n",
       "                                     19           Transfix - Online featurization and prediction...   \n",
       "                                     20           Habitat - Time-series feature engineering  \\nU...   \n",
       "                                     21                                    KI - Feature engineering   \n",
       "\n",
       "                                                                                          embedding  \\\n",
       "document_id                          chunk_index                                                      \n",
       "b5d59eed-cab7-4ed4-909f-940b4f22c22c 0            [-0.02026444487273693, 0.009682427160441875, 0...   \n",
       "                                     1            [-0.03777306526899338, 0.0073875077068805695, ...   \n",
       "                                     2            [-0.01697148010134697, 0.02586626075208187, 0....   \n",
       "                                     3            [-0.018855366855859756, -0.00249655288644135, ...   \n",
       "                                     4            [-0.047300878912210464, 0.031106166541576385, ...   \n",
       "                                     5            [-0.013192436657845974, -0.025940468534827232,...   \n",
       "                                     6            [-0.064208984375, -0.01950087398290634, 0.0302...   \n",
       "                                     7            [-0.01695844531059265, -0.023515893146395683, ...   \n",
       "                                     8            [0.0007336082635447383, 0.012326962314546108, ...   \n",
       "                                     9            [0.007231128867715597, 0.018477486446499825, 0...   \n",
       "                                     10           [-0.04146569222211838, 0.01626516319811344, 0....   \n",
       "                                     11           [-0.03983769938349724, 0.01923440210521221, 0....   \n",
       "                                     12           [-0.04583346098661423, 0.022130435332655907, 0...   \n",
       "                                     13           [-0.04830301180481911, 0.021801836788654327, 0...   \n",
       "                                     14           [-0.041252683848142624, 0.04047299176454544, 0...   \n",
       "                                     15           [-0.012794064357876778, 0.018729455769062042, ...   \n",
       "                                     16           [-0.045535918325185776, 0.034158188849687576, ...   \n",
       "                                     17           [0.0031941100023686886, 0.030165577307343483, ...   \n",
       "                                     18           [-0.00601744232699275, 0.008543377742171288, 0...   \n",
       "                                     19           [-0.0015230675926432014, 0.010722102597355843,...   \n",
       "                                     20           [0.009830841794610023, 0.004250616300851107, 0...   \n",
       "                                     21           [-0.010076257400214672, 0.01848677173256874, 0...   \n",
       "\n",
       "                                                                                       metadata  \n",
       "document_id                          chunk_index                                                 \n",
       "b5d59eed-cab7-4ed4-909f-940b4f22c22c 0                     {'Header 1': 'Welcome to Hamilton¶'}  \n",
       "                                     1                     {'Header 1': 'Welcome to Hamilton¶'}  \n",
       "                                     2                     {'Header 1': 'Welcome to Hamilton¶'}  \n",
       "                                     3            {'Header 1': 'Why should you use Hamilton?¶'}  \n",
       "                                     4            {'Header 1': 'Why should you use Hamilton?¶'}  \n",
       "                                     5            {'Header 1': 'Why should you use Hamilton?¶'}  \n",
       "                                     6            {'Header 1': 'Why should you use Hamilton?¶'}  \n",
       "                                     7            {'Header 1': 'Why should you use Hamilton?¶'}  \n",
       "                                     8            {'Header 1': 'Why should you use Hamilton?¶'}  \n",
       "                                     9            {'Header 1': 'Why should you use Hamilton?¶'}  \n",
       "                                     10           {'Header 1': 'Why should you use Hamilton?¶'}  \n",
       "                                     11           {'Header 1': 'Why should you use Hamilton?¶'}  \n",
       "                                     12           {'Header 1': 'Why should you use Hamilton?¶'}  \n",
       "                                     13                  {'Header 1': 'Architecture Overview¶'}  \n",
       "                                     14                  {'Header 1': 'Architecture Overview¶'}  \n",
       "                                     15                  {'Header 1': 'Architecture Overview¶'}  \n",
       "                                     16                  {'Header 1': 'Architecture Overview¶'}  \n",
       "                                     17                  {'Header 1': 'Architecture Overview¶'}  \n",
       "                                     18                 {'Header 1': 'Who is using Hamilton?¶'}  \n",
       "                                     19                 {'Header 1': 'Who is using Hamilton?¶'}  \n",
       "                                     20                 {'Header 1': 'Who is using Hamilton?¶'}  \n",
       "                                     21                 {'Header 1': 'Who is using Hamilton?¶'}  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from hamilton import driver\n",
    "# Note: we don't import doc_pipeline because it was being created via the magic & cells above.\n",
    "\n",
    "# create the driver\n",
    "pipeline_driver = driver.Builder().with_modules(doc_pipeline).build()\n",
    "\n",
    "# execute the pipeline for the given URL \n",
    "results = pipeline_driver.execute([\"store\"], inputs={\"url\": \"https://hamilton.apache.org/\"})\n",
    "\n",
    "# show the dataframe for this document\n",
    "results[\"store\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1ade834",
   "metadata": {},
   "source": [
    "------\n",
    "\n",
    "# Appendix\n",
    "\n",
    "Here are some extras to help guide you on your next steps.\n",
    "\n",
    "1. Processing many documents -- some discuss here on scaling this pipeline.\n",
    "2. How did the Hamilton driver know what our pipeline was? -- An explanation about how the jupyter magic with Hamilton operated here.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "472b0094",
   "metadata": {},
   "source": [
    "## 1. Processing many documents\n",
    "\n",
    "The example above processes a single document. But what if you had a lot more to process? What would your options be?\n",
    "\n",
    "### Option 1: For loop\n",
    "One could do the following with our example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "4223c24f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>url</th>\n",
       "      <th>text</th>\n",
       "      <th>embedding</th>\n",
       "      <th>metadata</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>document_id</th>\n",
       "      <th>chunk_index</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"22\" valign=\"top\">20a0bb13-8719-42f4-8e14-738818eeb332</th>\n",
       "      <th>0</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Hamilton is a general-purpose framework to wri...</td>\n",
       "      <td>[-0.02026444487273693, 0.009682427160441875, 0...</td>\n",
       "      <td>{'Header 1': 'Welcome to Hamilton¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Directed Acyclic Graph (DAG) that can be execu...</td>\n",
       "      <td>[-0.03777306526899338, 0.0073875077068805695, ...</td>\n",
       "      <td>{'Header 1': 'Welcome to Hamilton¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>The ABC of Hamilton¶</td>\n",
       "      <td>[-0.01697148010134697, 0.02586626075208187, 0....</td>\n",
       "      <td>{'Header 1': 'Welcome to Hamilton¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Facilitate collaboration. By focusing on funct...</td>\n",
       "      <td>[-0.018855366855859756, -0.00249655288644135, ...</td>\n",
       "      <td>{'Header 1': 'Why should you use Hamilton?¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Visualizations can be generated directly from ...</td>\n",
       "      <td>[-0.047300878912210464, 0.031106166541576385, ...</td>\n",
       "      <td>{'Header 1': 'Why should you use Hamilton?¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Reduce development time. Hamilton dataflows ar...</td>\n",
       "      <td>[-0.013192436657845974, -0.025940468534827232,...</td>\n",
       "      <td>{'Header 1': 'Why should you use Hamilton?¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Hamilton Hub.</td>\n",
       "      <td>[-0.064208984375, -0.01950087398290634, 0.0302...</td>\n",
       "      <td>{'Header 1': 'Why should you use Hamilton?¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Own your platform. Hamilton helps you integrat...</td>\n",
       "      <td>[-0.01695844531059265, -0.023515893146395683, ...</td>\n",
       "      <td>{'Header 1': 'Why should you use Hamilton?¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Scale your dataflow. Hamilton separates transf...</td>\n",
       "      <td>[0.0007336082635447383, 0.012326962314546108, ...</td>\n",
       "      <td>{'Header 1': 'Why should you use Hamilton?¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>enterprise data workloads.</td>\n",
       "      <td>[0.007231128867715597, 0.018477486446499825, 0...</td>\n",
       "      <td>{'Header 1': 'Why should you use Hamilton?¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Here’s a quick overview of benefits that Hamil...</td>\n",
       "      <td>[-0.04146569222211838, 0.01626516319811344, 0....</td>\n",
       "      <td>{'Header 1': 'Why should you use Hamilton?¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Can visualize lineage easily  \\n✅  \\n❌  \\n❌  \\...</td>\n",
       "      <td>[-0.03983769938349724, 0.01923440210521221, 0....</td>\n",
       "      <td>{'Header 1': 'Why should you use Hamilton?¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Runs anywhere python runs  \\n✅  \\n❌  \\n❌  \\n❌ ...</td>\n",
       "      <td>[-0.04583346098661423, 0.022130435332655907, 0...</td>\n",
       "      <td>{'Header 1': 'Why should you use Hamilton?¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>The following diagram gives a simplified overv...</td>\n",
       "      <td>[-0.04830301180481911, 0.021801836788654327, 0...</td>\n",
       "      <td>{'Header 1': 'Architecture Overview¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Functions &amp; Module. Transformations are regula...</td>\n",
       "      <td>[-0.041252683848142624, 0.04047299176454544, 0...</td>\n",
       "      <td>{'Header 1': 'Architecture Overview¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Driver &amp; FunctionGraph. The Driver will automa...</td>\n",
       "      <td>[-0.012794064357876778, 0.018729455769062042, ...</td>\n",
       "      <td>{'Header 1': 'Architecture Overview¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Visualization. The FunctionGraph can be visual...</td>\n",
       "      <td>[-0.045535918325185776, 0.034158188849687576, ...</td>\n",
       "      <td>{'Header 1': 'Architecture Overview¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Execution. When requesting variables, the Driv...</td>\n",
       "      <td>[0.0031941100023686886, 0.030165577307343483, ...</td>\n",
       "      <td>{'Header 1': 'Architecture Overview¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Multiple companies are doing cool stuff with H...</td>\n",
       "      <td>[-0.00601744232699275, 0.008543377742171288, 0...</td>\n",
       "      <td>{'Header 1': 'Who is using Hamilton?¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Transfix - Online featurization and prediction...</td>\n",
       "      <td>[-0.0015230675926432014, 0.010722102597355843,...</td>\n",
       "      <td>{'Header 1': 'Who is using Hamilton?¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Habitat - Time-series feature engineering  \\nU...</td>\n",
       "      <td>[0.009830841794610023, 0.004250616300851107, 0...</td>\n",
       "      <td>{'Header 1': 'Who is using Hamilton?¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>KI - Feature engineering</td>\n",
       "      <td>[-0.010076257400214672, 0.01848677173256874, 0...</td>\n",
       "      <td>{'Header 1': 'Who is using Hamilton?¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"27\" valign=\"top\">7f106c7a-d272-4584-8eb5-ada419acbd39</th>\n",
       "      <th>0</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>Hamilton comes with a fully open-source UI tha...</td>\n",
       "      <td>[-0.033495984971523285, 0.02976820431649685, 0...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>A feature/artifact catalog for browsing/connec...</td>\n",
       "      <td>[-0.029500646516680717, 0.027440618723630905, ...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>A project explorer for viewing curating projec...</td>\n",
       "      <td>[-0.034877486526966095, -0.015646569430828094,...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>If you’d like a video walkthrough on getting s...</td>\n",
       "      <td>[-0.04170287773013115, 0.01874636672437191, -0...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>The Hamilton UI is contained within a set of d...</td>\n",
       "      <td>[-0.06400912255048752, -0.009062161669135094, ...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>To run this, you’ll need:  \\nDocker installed ...</td>\n",
       "      <td>[-0.021686555817723274, 0.03889773041009903, 0...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>Start by ensuring that the hamilton repository...</td>\n",
       "      <td>[-0.04604955390095711, 0.027341920882463455, 0...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>./deployment/run.sh  \\nThis will do the follow...</td>\n",
       "      <td>[-0.014610188081860542, -0.008541340939700603,...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>This takes a bit of time! So be patient. The s...</td>\n",
       "      <td>[0.005524422507733107, -0.007484701462090015, ...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>If building the Docker containers from scratch...</td>\n",
       "      <td>[0.008969233371317387, 0.012474973686039448, 0...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td># from the hamilton/ui directory ./deployment/...</td>\n",
       "      <td>[-0.025990990921854973, 0.0099107064306736, 0....</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>Now that you have your server running, you can...</td>\n",
       "      <td>[-0.019784538075327873, 0.004920551087707281, ...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>Then, navigate to the project page (dashboard/...</td>\n",
       "      <td>[-0.012636549770832062, -0.014818817377090454,...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>from hamilton_sdk import adapters tracker = ad...</td>\n",
       "      <td>[0.0016246301820501685, -0.014947794377803802,...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>driver.Builder() .with_config(your_config) .wi...</td>\n",
       "      <td>[0.02870572917163372, 0.0004390304966364056, 0...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>Then run your DAG, and follow the links in the...</td>\n",
       "      <td>[-0.05947300046682358, 0.01847035251557827, -0...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td># we assume you're in the Hamilton repository ...</td>\n",
       "      <td>[-0.029487399384379387, -0.01613418199121952, ...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>--email &lt;email&gt; --project_id &lt;project_id&gt;</td>\n",
       "      <td>[0.01633821241557598, -0.01278590690344572, 0....</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>You should see links in the logs to the UI, wh...</td>\n",
       "      <td>[-0.03280032426118851, 0.024312302470207214, -...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>Once you get to the UI, you can navigate to th...</td>\n",
       "      <td>[-0.013666111044585705, -0.000534116174094379,...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>See below for a few screenshots of the UI.</td>\n",
       "      <td>[-0.08185029029846191, 0.0024983373004943132, ...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>The UI has the following features:</td>\n",
       "      <td>[-0.0673123225569725, 0.02698035165667534, 0.0...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Feat...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>Select DAG versions to compare + visualize.</td>\n",
       "      <td>[-0.02967393398284912, -0.04668116569519043, 0...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Feat...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>View functions, nodes, and assets across a his...</td>\n",
       "      <td>[-0.03586745634675026, 0.022120537236332893, 0...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Feat...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>View DAG shapes + code:</td>\n",
       "      <td>[-0.048070456832647324, -0.0048097106628119946...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Feat...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>View a history of runs, telemetry on runs/comp...</td>\n",
       "      <td>[-0.014719597063958645, 0.01228130329400301, 0...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Feat...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>Please reach out to us if you want to deploy o...</td>\n",
       "      <td>[0.013612204231321812, -0.003925923258066177, ...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Self...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                url  \\\n",
       "document_id                          chunk_index                                                      \n",
       "20a0bb13-8719-42f4-8e14-738818eeb332 0                      https://hamilton.dagworks.io/en/latest/   \n",
       "                                     1                      https://hamilton.dagworks.io/en/latest/   \n",
       "                                     2                      https://hamilton.dagworks.io/en/latest/   \n",
       "                                     3                      https://hamilton.dagworks.io/en/latest/   \n",
       "                                     4                      https://hamilton.dagworks.io/en/latest/   \n",
       "                                     5                      https://hamilton.dagworks.io/en/latest/   \n",
       "                                     6                      https://hamilton.dagworks.io/en/latest/   \n",
       "                                     7                      https://hamilton.dagworks.io/en/latest/   \n",
       "                                     8                      https://hamilton.dagworks.io/en/latest/   \n",
       "                                     9                      https://hamilton.dagworks.io/en/latest/   \n",
       "                                     10                     https://hamilton.dagworks.io/en/latest/   \n",
       "                                     11                     https://hamilton.dagworks.io/en/latest/   \n",
       "                                     12                     https://hamilton.dagworks.io/en/latest/   \n",
       "                                     13                     https://hamilton.dagworks.io/en/latest/   \n",
       "                                     14                     https://hamilton.dagworks.io/en/latest/   \n",
       "                                     15                     https://hamilton.dagworks.io/en/latest/   \n",
       "                                     16                     https://hamilton.dagworks.io/en/latest/   \n",
       "                                     17                     https://hamilton.dagworks.io/en/latest/   \n",
       "                                     18                     https://hamilton.dagworks.io/en/latest/   \n",
       "                                     19                     https://hamilton.dagworks.io/en/latest/   \n",
       "                                     20                     https://hamilton.dagworks.io/en/latest/   \n",
       "                                     21                     https://hamilton.dagworks.io/en/latest/   \n",
       "7f106c7a-d272-4584-8eb5-ada419acbd39 0            https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     1            https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     2            https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     3            https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     4            https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     5            https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     6            https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     7            https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     8            https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     9            https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     10           https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     11           https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     12           https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     13           https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     14           https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     15           https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     16           https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     17           https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     18           https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     19           https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     20           https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     21           https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     22           https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     23           https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     24           https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     25           https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     26           https://hamilton.dagworks.io/en/latest/concept...   \n",
       "\n",
       "                                                                                               text  \\\n",
       "document_id                          chunk_index                                                      \n",
       "20a0bb13-8719-42f4-8e14-738818eeb332 0            Hamilton is a general-purpose framework to wri...   \n",
       "                                     1            Directed Acyclic Graph (DAG) that can be execu...   \n",
       "                                     2                                         The ABC of Hamilton¶   \n",
       "                                     3            Facilitate collaboration. By focusing on funct...   \n",
       "                                     4            Visualizations can be generated directly from ...   \n",
       "                                     5            Reduce development time. Hamilton dataflows ar...   \n",
       "                                     6                                                Hamilton Hub.   \n",
       "                                     7            Own your platform. Hamilton helps you integrat...   \n",
       "                                     8            Scale your dataflow. Hamilton separates transf...   \n",
       "                                     9                                   enterprise data workloads.   \n",
       "                                     10           Here’s a quick overview of benefits that Hamil...   \n",
       "                                     11           Can visualize lineage easily  \\n✅  \\n❌  \\n❌  \\...   \n",
       "                                     12           Runs anywhere python runs  \\n✅  \\n❌  \\n❌  \\n❌ ...   \n",
       "                                     13           The following diagram gives a simplified overv...   \n",
       "                                     14           Functions & Module. Transformations are regula...   \n",
       "                                     15           Driver & FunctionGraph. The Driver will automa...   \n",
       "                                     16           Visualization. The FunctionGraph can be visual...   \n",
       "                                     17           Execution. When requesting variables, the Driv...   \n",
       "                                     18           Multiple companies are doing cool stuff with H...   \n",
       "                                     19           Transfix - Online featurization and prediction...   \n",
       "                                     20           Habitat - Time-series feature engineering  \\nU...   \n",
       "                                     21                                    KI - Feature engineering   \n",
       "7f106c7a-d272-4584-8eb5-ada419acbd39 0            Hamilton comes with a fully open-source UI tha...   \n",
       "                                     1            A feature/artifact catalog for browsing/connec...   \n",
       "                                     2            A project explorer for viewing curating projec...   \n",
       "                                     3            If you’d like a video walkthrough on getting s...   \n",
       "                                     4            The Hamilton UI is contained within a set of d...   \n",
       "                                     5            To run this, you’ll need:  \\nDocker installed ...   \n",
       "                                     6            Start by ensuring that the hamilton repository...   \n",
       "                                     7            ./deployment/run.sh  \\nThis will do the follow...   \n",
       "                                     8            This takes a bit of time! So be patient. The s...   \n",
       "                                     9            If building the Docker containers from scratch...   \n",
       "                                     10           # from the hamilton/ui directory ./deployment/...   \n",
       "                                     11           Now that you have your server running, you can...   \n",
       "                                     12           Then, navigate to the project page (dashboard/...   \n",
       "                                     13           from hamilton_sdk import adapters tracker = ad...   \n",
       "                                     14           driver.Builder() .with_config(your_config) .wi...   \n",
       "                                     15           Then run your DAG, and follow the links in the...   \n",
       "                                     16           # we assume you're in the Hamilton repository ...   \n",
       "                                     17                   --email <email> --project_id <project_id>   \n",
       "                                     18           You should see links in the logs to the UI, wh...   \n",
       "                                     19           Once you get to the UI, you can navigate to th...   \n",
       "                                     20                  See below for a few screenshots of the UI.   \n",
       "                                     21                          The UI has the following features:   \n",
       "                                     22                 Select DAG versions to compare + visualize.   \n",
       "                                     23           View functions, nodes, and assets across a his...   \n",
       "                                     24                                     View DAG shapes + code:   \n",
       "                                     25           View a history of runs, telemetry on runs/comp...   \n",
       "                                     26           Please reach out to us if you want to deploy o...   \n",
       "\n",
       "                                                                                          embedding  \\\n",
       "document_id                          chunk_index                                                      \n",
       "20a0bb13-8719-42f4-8e14-738818eeb332 0            [-0.02026444487273693, 0.009682427160441875, 0...   \n",
       "                                     1            [-0.03777306526899338, 0.0073875077068805695, ...   \n",
       "                                     2            [-0.01697148010134697, 0.02586626075208187, 0....   \n",
       "                                     3            [-0.018855366855859756, -0.00249655288644135, ...   \n",
       "                                     4            [-0.047300878912210464, 0.031106166541576385, ...   \n",
       "                                     5            [-0.013192436657845974, -0.025940468534827232,...   \n",
       "                                     6            [-0.064208984375, -0.01950087398290634, 0.0302...   \n",
       "                                     7            [-0.01695844531059265, -0.023515893146395683, ...   \n",
       "                                     8            [0.0007336082635447383, 0.012326962314546108, ...   \n",
       "                                     9            [0.007231128867715597, 0.018477486446499825, 0...   \n",
       "                                     10           [-0.04146569222211838, 0.01626516319811344, 0....   \n",
       "                                     11           [-0.03983769938349724, 0.01923440210521221, 0....   \n",
       "                                     12           [-0.04583346098661423, 0.022130435332655907, 0...   \n",
       "                                     13           [-0.04830301180481911, 0.021801836788654327, 0...   \n",
       "                                     14           [-0.041252683848142624, 0.04047299176454544, 0...   \n",
       "                                     15           [-0.012794064357876778, 0.018729455769062042, ...   \n",
       "                                     16           [-0.045535918325185776, 0.034158188849687576, ...   \n",
       "                                     17           [0.0031941100023686886, 0.030165577307343483, ...   \n",
       "                                     18           [-0.00601744232699275, 0.008543377742171288, 0...   \n",
       "                                     19           [-0.0015230675926432014, 0.010722102597355843,...   \n",
       "                                     20           [0.009830841794610023, 0.004250616300851107, 0...   \n",
       "                                     21           [-0.010076257400214672, 0.01848677173256874, 0...   \n",
       "7f106c7a-d272-4584-8eb5-ada419acbd39 0            [-0.033495984971523285, 0.02976820431649685, 0...   \n",
       "                                     1            [-0.029500646516680717, 0.027440618723630905, ...   \n",
       "                                     2            [-0.034877486526966095, -0.015646569430828094,...   \n",
       "                                     3            [-0.04170287773013115, 0.01874636672437191, -0...   \n",
       "                                     4            [-0.06400912255048752, -0.009062161669135094, ...   \n",
       "                                     5            [-0.021686555817723274, 0.03889773041009903, 0...   \n",
       "                                     6            [-0.04604955390095711, 0.027341920882463455, 0...   \n",
       "                                     7            [-0.014610188081860542, -0.008541340939700603,...   \n",
       "                                     8            [0.005524422507733107, -0.007484701462090015, ...   \n",
       "                                     9            [0.008969233371317387, 0.012474973686039448, 0...   \n",
       "                                     10           [-0.025990990921854973, 0.0099107064306736, 0....   \n",
       "                                     11           [-0.019784538075327873, 0.004920551087707281, ...   \n",
       "                                     12           [-0.012636549770832062, -0.014818817377090454,...   \n",
       "                                     13           [0.0016246301820501685, -0.014947794377803802,...   \n",
       "                                     14           [0.02870572917163372, 0.0004390304966364056, 0...   \n",
       "                                     15           [-0.05947300046682358, 0.01847035251557827, -0...   \n",
       "                                     16           [-0.029487399384379387, -0.01613418199121952, ...   \n",
       "                                     17           [0.01633821241557598, -0.01278590690344572, 0....   \n",
       "                                     18           [-0.03280032426118851, 0.024312302470207214, -...   \n",
       "                                     19           [-0.013666111044585705, -0.000534116174094379,...   \n",
       "                                     20           [-0.08185029029846191, 0.0024983373004943132, ...   \n",
       "                                     21           [-0.0673123225569725, 0.02698035165667534, 0.0...   \n",
       "                                     22           [-0.02967393398284912, -0.04668116569519043, 0...   \n",
       "                                     23           [-0.03586745634675026, 0.022120537236332893, 0...   \n",
       "                                     24           [-0.048070456832647324, -0.0048097106628119946...   \n",
       "                                     25           [-0.014719597063958645, 0.01228130329400301, 0...   \n",
       "                                     26           [0.013612204231321812, -0.003925923258066177, ...   \n",
       "\n",
       "                                                                                           metadata  \n",
       "document_id                          chunk_index                                                     \n",
       "20a0bb13-8719-42f4-8e14-738818eeb332 0                         {'Header 1': 'Welcome to Hamilton¶'}  \n",
       "                                     1                         {'Header 1': 'Welcome to Hamilton¶'}  \n",
       "                                     2                         {'Header 1': 'Welcome to Hamilton¶'}  \n",
       "                                     3                {'Header 1': 'Why should you use Hamilton?¶'}  \n",
       "                                     4                {'Header 1': 'Why should you use Hamilton?¶'}  \n",
       "                                     5                {'Header 1': 'Why should you use Hamilton?¶'}  \n",
       "                                     6                {'Header 1': 'Why should you use Hamilton?¶'}  \n",
       "                                     7                {'Header 1': 'Why should you use Hamilton?¶'}  \n",
       "                                     8                {'Header 1': 'Why should you use Hamilton?¶'}  \n",
       "                                     9                {'Header 1': 'Why should you use Hamilton?¶'}  \n",
       "                                     10               {'Header 1': 'Why should you use Hamilton?¶'}  \n",
       "                                     11               {'Header 1': 'Why should you use Hamilton?¶'}  \n",
       "                                     12               {'Header 1': 'Why should you use Hamilton?¶'}  \n",
       "                                     13                      {'Header 1': 'Architecture Overview¶'}  \n",
       "                                     14                      {'Header 1': 'Architecture Overview¶'}  \n",
       "                                     15                      {'Header 1': 'Architecture Overview¶'}  \n",
       "                                     16                      {'Header 1': 'Architecture Overview¶'}  \n",
       "                                     17                      {'Header 1': 'Architecture Overview¶'}  \n",
       "                                     18                     {'Header 1': 'Who is using Hamilton?¶'}  \n",
       "                                     19                     {'Header 1': 'Who is using Hamilton?¶'}  \n",
       "                                     20                     {'Header 1': 'Who is using Hamilton?¶'}  \n",
       "                                     21                     {'Header 1': 'Who is using Hamilton?¶'}  \n",
       "7f106c7a-d272-4584-8eb5-ada419acbd39 0                                 {'Header 1': 'Hamilton UI¶'}  \n",
       "                                     1                                 {'Header 1': 'Hamilton UI¶'}  \n",
       "                                     2                                 {'Header 1': 'Hamilton UI¶'}  \n",
       "                                     3            {'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...  \n",
       "                                     4            {'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...  \n",
       "                                     5            {'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...  \n",
       "                                     6            {'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...  \n",
       "                                     7            {'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...  \n",
       "                                     8            {'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...  \n",
       "                                     9            {'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...  \n",
       "                                     10           {'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...  \n",
       "                                     11           {'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...  \n",
       "                                     12           {'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...  \n",
       "                                     13           {'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...  \n",
       "                                     14           {'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...  \n",
       "                                     15           {'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...  \n",
       "                                     16           {'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...  \n",
       "                                     17           {'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...  \n",
       "                                     18           {'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...  \n",
       "                                     19           {'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...  \n",
       "                                     20           {'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...  \n",
       "                                     21           {'Header 1': 'Hamilton UI¶', 'Header 2': 'Feat...  \n",
       "                                     22           {'Header 1': 'Hamilton UI¶', 'Header 2': 'Feat...  \n",
       "                                     23           {'Header 1': 'Hamilton UI¶', 'Header 2': 'Feat...  \n",
       "                                     24           {'Header 1': 'Hamilton UI¶', 'Header 2': 'Feat...  \n",
       "                                     25           {'Header 1': 'Hamilton UI¶', 'Header 2': 'Feat...  \n",
       "                                     26           {'Header 1': 'Hamilton UI¶', 'Header 2': 'Self...  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "urls = [\"https://hamilton.apache.org/\", \"https://hamilton.apache.org/concepts/ui\"] # some list of URLs\n",
    "dataframes = []\n",
    "for url in urls:\n",
    "    results = pipeline_driver.execute([\"store\"], inputs={\"url\": url})\n",
    "    dataframes.append(results[\"store\"])\n",
    "# concatenate the dataframes into one big document\n",
    "pd.concat(dataframes)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3e569aa",
   "metadata": {},
   "source": [
    "We could parallelize this above loop by using a multi-processing/multi-threading (because it's largely I/O bound).\n",
    "Or use Hamilton's parallelization capabilties (see next section).\n",
    "\n",
    "### Options 2: Use Parallel & Collect to parallelize\n",
    "We need to add two more functions that will \"book end\" the pipeline. We won't dive into it, but [more documentation can be found here](https://hamilton.apache.org/concepts/parallel-task/), that explains that Hamilton can delegate DAG execution in various ways. Below we'll show how it plugs into multi-threading. There are other way to execute, e.g. via Ray, or Dask."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "1f07c96f",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%cell_to_module parallel_pipeline\n",
    "# we create a new module called parallel_pipeline\n",
    "from hamilton.htypes import Collect, Parallelizable\n",
    "import pandas as pd\n",
    "\n",
    "def url(urls: list[str]) -> Parallelizable[str]:\n",
    "    \"\"\"This starts a parallel block -- anything that requires `url` will be parallelized.\"\"\"\n",
    "    for url in urls:\n",
    "        yield url\n",
    "\n",
    "def collected_dataframes(store: Collect[pd.DataFrame]) -> pd.DataFrame:\n",
    "    \"\"\"This ends the parallel block. We will collect the result for every `store`.\"\"\"\n",
    "    return pd.concat(store)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "d7e74ceb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 10.0.1 (20240210.2158)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"1187pt\" height=\"503pt\"\n",
       " viewBox=\"0.00 0.00 1186.70 502.80\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 498.8)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-498.8 1182.7,-498.8 1182.7,4 -4,4\"/>\n",
       "<g id=\"clust1\" class=\"cluster\">\n",
       "<title>cluster__legend</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" points=\"8,-230.8 8,-486.8 97.85,-486.8 97.85,-230.8 8,-230.8\"/>\n",
       "<text text-anchor=\"middle\" x=\"52.92\" y=\"-469.5\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Legend</text>\n",
       "</g>\n",
       "<!-- chunked_document -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>chunked_document</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M658.9,-145.6C658.9,-145.6 533.05,-145.6 533.05,-145.6 527.05,-145.6 521.05,-139.6 521.05,-133.6 521.05,-133.6 521.05,-94 521.05,-94 521.05,-88 527.05,-82 533.05,-82 533.05,-82 658.9,-82 658.9,-82 664.9,-82 670.9,-88 670.9,-94 670.9,-94 670.9,-133.6 670.9,-133.6 670.9,-139.6 664.9,-145.6 658.9,-145.6\"/>\n",
       "<text text-anchor=\"start\" x=\"531.85\" y=\"-122.5\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">chunked_document</text>\n",
       "<text text-anchor=\"start\" x=\"564.85\" y=\"-94.5\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">Document</text>\n",
       "</g>\n",
       "<!-- embedded_document -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>embedded_document</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M850.5,-186.6C850.5,-186.6 711.9,-186.6 711.9,-186.6 705.9,-186.6 699.9,-180.6 699.9,-174.6 699.9,-174.6 699.9,-135 699.9,-135 699.9,-129 705.9,-123 711.9,-123 711.9,-123 850.5,-123 850.5,-123 856.5,-123 862.5,-129 862.5,-135 862.5,-135 862.5,-174.6 862.5,-174.6 862.5,-180.6 856.5,-186.6 850.5,-186.6\"/>\n",
       "<text text-anchor=\"start\" x=\"710.7\" y=\"-163.5\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">embedded_document</text>\n",
       "<text text-anchor=\"start\" x=\"750.08\" y=\"-135.5\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">Document</text>\n",
       "</g>\n",
       "<!-- chunked_document&#45;&gt;embedded_document -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>chunked_document&#45;&gt;embedded_document</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M671.24,-130.42C676.92,-131.69 682.69,-132.98 688.47,-134.27\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"687.55,-137.65 698.07,-136.42 689.08,-130.82 687.55,-137.65\"/>\n",
       "</g>\n",
       "<!-- store -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>store</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M969.35,-186.6C969.35,-186.6 903.5,-186.6 903.5,-186.6 897.5,-186.6 891.5,-180.6 891.5,-174.6 891.5,-174.6 891.5,-135 891.5,-135 891.5,-129 897.5,-123 903.5,-123 903.5,-123 969.35,-123 969.35,-123 975.35,-123 981.35,-129 981.35,-135 981.35,-135 981.35,-174.6 981.35,-174.6 981.35,-180.6 975.35,-186.6 969.35,-186.6\"/>\n",
       "<text text-anchor=\"start\" x=\"919.93\" y=\"-163.5\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">store</text>\n",
       "<text text-anchor=\"start\" x=\"902.3\" y=\"-135.5\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">DataFrame</text>\n",
       "</g>\n",
       "<!-- embedded_document&#45;&gt;store -->\n",
       "<g id=\"edge10\" class=\"edge\">\n",
       "<title>embedded_document&#45;&gt;store</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M862.85,-154.8C868.67,-154.8 874.46,-154.8 880.09,-154.8\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"879.84,-158.3 889.84,-154.8 879.84,-151.3 879.84,-158.3\"/>\n",
       "</g>\n",
       "<!-- html_regex -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>html_regex</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M222.32,-315.6C222.32,-315.6 151.97,-315.6 151.97,-315.6 145.97,-315.6 139.97,-309.6 139.97,-303.6 139.97,-303.6 139.97,-264 139.97,-264 139.97,-258 145.97,-252 151.97,-252 151.97,-252 222.32,-252 222.32,-252 228.32,-252 234.32,-258 234.32,-264 234.32,-264 234.32,-303.6 234.32,-303.6 234.32,-309.6 228.32,-315.6 222.32,-315.6\"/>\n",
       "<text text-anchor=\"start\" x=\"150.77\" y=\"-292.5\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">html_regex</text>\n",
       "<text text-anchor=\"start\" x=\"179.65\" y=\"-264.5\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- raw_document -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>raw_document</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M435.43,-229.6C435.43,-229.6 341.07,-229.6 341.07,-229.6 335.07,-229.6 329.07,-223.6 329.07,-217.6 329.07,-217.6 329.07,-178 329.07,-178 329.07,-172 335.07,-166 341.07,-166 341.07,-166 435.43,-166 435.43,-166 441.43,-166 447.43,-172 447.43,-178 447.43,-178 447.43,-217.6 447.43,-217.6 447.43,-223.6 441.43,-229.6 435.43,-229.6\"/>\n",
       "<text text-anchor=\"start\" x=\"339.88\" y=\"-206.5\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">raw_document</text>\n",
       "<text text-anchor=\"start\" x=\"357.12\" y=\"-178.5\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">Document</text>\n",
       "</g>\n",
       "<!-- html_regex&#45;&gt;raw_document -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>html_regex&#45;&gt;raw_document</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M234.58,-263.75C259.44,-253.01 290.55,-239.57 318.19,-227.63\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"319.41,-230.91 327.21,-223.74 316.64,-224.49 319.41,-230.91\"/>\n",
       "</g>\n",
       "<!-- url -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>url</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"#56e39f\" d=\"M226.07,-229.6C226.07,-229.6 148.22,-229.6 148.22,-229.6 142.22,-229.6 136.22,-223.6 136.22,-217.6 136.22,-217.6 136.22,-178 136.22,-178 136.22,-172 142.22,-166 148.22,-166 148.22,-166 226.07,-166 226.07,-166 232.07,-166 238.07,-172 238.07,-178 238.07,-178 238.07,-217.6 238.07,-217.6 238.07,-223.6 232.07,-229.6 226.07,-229.6\"/>\n",
       "<path fill=\"none\" stroke=\"#56e39f\" d=\"M230.07,-233.6C230.07,-233.6 144.22,-233.6 144.22,-233.6 138.22,-233.6 132.22,-227.6 132.22,-221.6 132.22,-221.6 132.22,-174 132.22,-174 132.22,-168 138.22,-162 144.22,-162 144.22,-162 230.07,-162 230.07,-162 236.07,-162 242.07,-168 242.07,-174 242.07,-174 242.07,-221.6 242.07,-221.6 242.07,-227.6 236.07,-233.6 230.07,-233.6\"/>\n",
       "<text text-anchor=\"start\" x=\"178.52\" y=\"-206.5\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">url</text>\n",
       "<text text-anchor=\"start\" x=\"147.02\" y=\"-178.5\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">Parallelizable</text>\n",
       "</g>\n",
       "<!-- url&#45;&gt;raw_document -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>url&#45;&gt;raw_document</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M242.55,-197.8C265.93,-197.8 293.51,-197.8 318.3,-197.8\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"318.39,-197.8 328.39,-202.3 323.72,-197.8 328.06,-197.8 328.06,-197.8 328.06,-197.8 323.72,-197.8 328.39,-193.3 318.39,-197.8\"/>\n",
       "</g>\n",
       "<!-- raw_document&#45;&gt;chunked_document -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>raw_document&#45;&gt;chunked_document</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M447.65,-173.95C467.06,-166.03 489.09,-157.04 510.08,-148.46\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"511.28,-151.75 519.22,-144.73 508.63,-145.27 511.28,-151.75\"/>\n",
       "</g>\n",
       "<!-- client -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>client</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M618.02,-227.6C618.02,-227.6 573.93,-227.6 573.93,-227.6 567.93,-227.6 561.93,-221.6 561.93,-215.6 561.93,-215.6 561.93,-176 561.93,-176 561.93,-170 567.93,-164 573.93,-164 573.93,-164 618.02,-164 618.02,-164 624.02,-164 630.02,-170 630.02,-176 630.02,-176 630.02,-215.6 630.02,-215.6 630.02,-221.6 624.02,-227.6 618.02,-227.6\"/>\n",
       "<text text-anchor=\"start\" x=\"578.35\" y=\"-204.5\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">client</text>\n",
       "<text text-anchor=\"start\" x=\"572.73\" y=\"-176.5\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">OpenAI</text>\n",
       "</g>\n",
       "<!-- client&#45;&gt;embedded_document -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>client&#45;&gt;embedded_document</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M630.16,-188.38C646.68,-184.68 667.55,-180.01 688.43,-175.33\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"689.2,-178.75 698.19,-173.15 687.67,-171.92 689.2,-178.75\"/>\n",
       "</g>\n",
       "<!-- text_chunker -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>text_chunker</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M480.05,-145.6C480.05,-145.6 296.45,-145.6 296.45,-145.6 290.45,-145.6 284.45,-139.6 284.45,-133.6 284.45,-133.6 284.45,-94 284.45,-94 284.45,-88 290.45,-82 296.45,-82 296.45,-82 480.05,-82 480.05,-82 486.05,-82 492.05,-88 492.05,-94 492.05,-94 492.05,-133.6 492.05,-133.6 492.05,-139.6 486.05,-145.6 480.05,-145.6\"/>\n",
       "<text text-anchor=\"start\" x=\"346.25\" y=\"-122.5\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">text_chunker</text>\n",
       "<text text-anchor=\"start\" x=\"295.25\" y=\"-94.5\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">RecursiveCharacterTextSplitter</text>\n",
       "</g>\n",
       "<!-- text_chunker&#45;&gt;chunked_document -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>text_chunker&#45;&gt;chunked_document</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M492.41,-113.8C498.07,-113.8 503.72,-113.8 509.31,-113.8\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"509.14,-117.3 519.14,-113.8 509.14,-110.3 509.14,-117.3\"/>\n",
       "</g>\n",
       "<!-- collected_dataframes -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>collected_dataframes</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"#ea5556\" d=\"M1162.7,-186.6C1162.7,-186.6 1026.35,-186.6 1026.35,-186.6 1020.35,-186.6 1014.35,-180.6 1014.35,-174.6 1014.35,-174.6 1014.35,-135 1014.35,-135 1014.35,-129 1020.35,-123 1026.35,-123 1026.35,-123 1162.7,-123 1162.7,-123 1168.7,-123 1174.7,-129 1174.7,-135 1174.7,-135 1174.7,-174.6 1174.7,-174.6 1174.7,-180.6 1168.7,-186.6 1162.7,-186.6\"/>\n",
       "<path fill=\"none\" stroke=\"#ea5556\" d=\"M1166.7,-190.6C1166.7,-190.6 1022.35,-190.6 1022.35,-190.6 1016.35,-190.6 1010.35,-184.6 1010.35,-178.6 1010.35,-178.6 1010.35,-131 1010.35,-131 1010.35,-125 1016.35,-119 1022.35,-119 1022.35,-119 1166.7,-119 1166.7,-119 1172.7,-119 1178.7,-125 1178.7,-131 1178.7,-131 1178.7,-178.6 1178.7,-178.6 1178.7,-184.6 1172.7,-190.6 1166.7,-190.6\"/>\n",
       "<text text-anchor=\"start\" x=\"1025.15\" y=\"-163.5\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">collected_dataframes</text>\n",
       "<text text-anchor=\"start\" x=\"1060.4\" y=\"-135.5\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">DataFrame</text>\n",
       "</g>\n",
       "<!-- store&#45;&gt;collected_dataframes -->\n",
       "<g id=\"edge11\" class=\"edge\">\n",
       "<title>store&#45;&gt;collected_dataframes</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M992.05,-154.8C994.25,-154.8 996.48,-154.8 998.73,-154.8\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"992.07,-154.8 982.07,-150.3 986.74,-154.8 982.41,-154.8 982.41,-154.8 982.41,-154.8 986.74,-154.8 982.07,-159.3 992.07,-154.8\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"998.57,-158.3 1008.57,-154.8 998.57,-151.3 998.57,-158.3\"/>\n",
       "</g>\n",
       "<!-- html_chunker -->\n",
       "<g id=\"node10\" class=\"node\">\n",
       "<title>html_chunker</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M461.3,-63.6C461.3,-63.6 315.2,-63.6 315.2,-63.6 309.2,-63.6 303.2,-57.6 303.2,-51.6 303.2,-51.6 303.2,-12 303.2,-12 303.2,-6 309.2,0 315.2,0 315.2,0 461.3,0 461.3,0 467.3,0 473.3,-6 473.3,-12 473.3,-12 473.3,-51.6 473.3,-51.6 473.3,-57.6 467.3,-63.6 461.3,-63.6\"/>\n",
       "<text text-anchor=\"start\" x=\"343.62\" y=\"-40.5\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">html_chunker</text>\n",
       "<text text-anchor=\"start\" x=\"314\" y=\"-12.5\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">HTMLHeaderTextSplitter</text>\n",
       "</g>\n",
       "<!-- html_chunker&#45;&gt;chunked_document -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>html_chunker&#45;&gt;chunked_document</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M470.26,-64.09C483.39,-69.32 497.02,-74.76 510.24,-80.03\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"508.74,-83.2 519.33,-83.65 511.33,-76.69 508.74,-83.2\"/>\n",
       "</g>\n",
       "<!-- _url_inputs -->\n",
       "<g id=\"node11\" class=\"node\">\n",
       "<title>_url_inputs</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"89.85,-220.1 16,-220.1 16,-175.5 89.85,-175.5 89.85,-220.1\"/>\n",
       "<text text-anchor=\"start\" x=\"30.8\" y=\"-192\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">urls</text>\n",
       "<text text-anchor=\"start\" x=\"58.55\" y=\"-192\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">list</text>\n",
       "</g>\n",
       "<!-- _url_inputs&#45;&gt;url -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>_url_inputs&#45;&gt;url</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M90.06,-197.8C99.55,-197.8 110.08,-197.8 120.6,-197.8\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"120.44,-201.3 130.44,-197.8 120.44,-194.3 120.44,-201.3\"/>\n",
       "</g>\n",
       "<!-- _text_chunker_inputs -->\n",
       "<g id=\"node12\" class=\"node\">\n",
       "<title>_text_chunker_inputs</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"255.45,-143.6 118.85,-143.6 118.85,-78 255.45,-78 255.45,-143.6\"/>\n",
       "<text text-anchor=\"start\" x=\"143.4\" y=\"-115.5\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">chunk_size</text>\n",
       "<text text-anchor=\"start\" x=\"226.52\" y=\"-115.5\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">int</text>\n",
       "<text text-anchor=\"start\" x=\"133.27\" y=\"-94.5\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">chunk_overlap</text>\n",
       "<text text-anchor=\"start\" x=\"226.52\" y=\"-94.5\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">int</text>\n",
       "</g>\n",
       "<!-- _text_chunker_inputs&#45;&gt;text_chunker -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>_text_chunker_inputs&#45;&gt;text_chunker</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M255.83,-111.82C261.4,-111.9 267.11,-111.99 272.89,-112.08\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"272.51,-115.57 282.56,-112.22 272.61,-108.57 272.51,-115.57\"/>\n",
       "</g>\n",
       "<!-- input -->\n",
       "<g id=\"node13\" class=\"node\">\n",
       "<title>input</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"79.92,-275.1 25.92,-275.1 25.92,-238.5 79.92,-238.5 79.92,-275.1\"/>\n",
       "<text text-anchor=\"middle\" x=\"52.92\" y=\"-251\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">input</text>\n",
       "</g>\n",
       "<!-- function -->\n",
       "<g id=\"node14\" class=\"node\">\n",
       "<title>function</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M75.35,-330.1C75.35,-330.1 30.5,-330.1 30.5,-330.1 24.5,-330.1 18.5,-324.1 18.5,-318.1 18.5,-318.1 18.5,-305.5 18.5,-305.5 18.5,-299.5 24.5,-293.5 30.5,-293.5 30.5,-293.5 75.35,-293.5 75.35,-293.5 81.35,-293.5 87.35,-299.5 87.35,-305.5 87.35,-305.5 87.35,-318.1 87.35,-318.1 87.35,-324.1 81.35,-330.1 75.35,-330.1\"/>\n",
       "<text text-anchor=\"middle\" x=\"52.92\" y=\"-306\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">function</text>\n",
       "</g>\n",
       "<!-- expand -->\n",
       "<g id=\"node15\" class=\"node\">\n",
       "<title>expand</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"#56e39f\" d=\"M73.85,-389.1C73.85,-389.1 32,-389.1 32,-389.1 26,-389.1 20,-383.1 20,-377.1 20,-377.1 20,-364.5 20,-364.5 20,-358.5 26,-352.5 32,-352.5 32,-352.5 73.85,-352.5 73.85,-352.5 79.85,-352.5 85.85,-358.5 85.85,-364.5 85.85,-364.5 85.85,-377.1 85.85,-377.1 85.85,-383.1 79.85,-389.1 73.85,-389.1\"/>\n",
       "<path fill=\"none\" stroke=\"#56e39f\" d=\"M77.85,-393.1C77.85,-393.1 28,-393.1 28,-393.1 22,-393.1 16,-387.1 16,-381.1 16,-381.1 16,-360.5 16,-360.5 16,-354.5 22,-348.5 28,-348.5 28,-348.5 77.85,-348.5 77.85,-348.5 83.85,-348.5 89.85,-354.5 89.85,-360.5 89.85,-360.5 89.85,-381.1 89.85,-381.1 89.85,-387.1 83.85,-393.1 77.85,-393.1\"/>\n",
       "<text text-anchor=\"middle\" x=\"52.92\" y=\"-365\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">expand</text>\n",
       "</g>\n",
       "<!-- collect -->\n",
       "<g id=\"node16\" class=\"node\">\n",
       "<title>collect</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"#ea5556\" d=\"M70.85,-452.1C70.85,-452.1 35,-452.1 35,-452.1 29,-452.1 23,-446.1 23,-440.1 23,-440.1 23,-427.5 23,-427.5 23,-421.5 29,-415.5 35,-415.5 35,-415.5 70.85,-415.5 70.85,-415.5 76.85,-415.5 82.85,-421.5 82.85,-427.5 82.85,-427.5 82.85,-440.1 82.85,-440.1 82.85,-446.1 76.85,-452.1 70.85,-452.1\"/>\n",
       "<path fill=\"none\" stroke=\"#ea5556\" d=\"M74.85,-456.1C74.85,-456.1 31,-456.1 31,-456.1 25,-456.1 19,-450.1 19,-444.1 19,-444.1 19,-423.5 19,-423.5 19,-417.5 25,-411.5 31,-411.5 31,-411.5 74.85,-411.5 74.85,-411.5 80.85,-411.5 86.85,-417.5 86.85,-423.5 86.85,-423.5 86.85,-444.1 86.85,-444.1 86.85,-450.1 80.85,-456.1 74.85,-456.1\"/>\n",
       "<text text-anchor=\"middle\" x=\"52.92\" y=\"-428\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">collect</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<hamilton.driver.Driver at 0x151a69100>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# now we create our parallel driver with both modules\n",
    "from hamilton.execution import executors\n",
    "from hamilton import driver\n",
    "\n",
    "parallel_dr = (\n",
    "    driver.Builder()\n",
    "    .with_modules(parallel_pipeline, doc_pipeline)  # we combine the two modules here\n",
    "    .enable_dynamic_execution(allow_experimental_mode=True)\n",
    "    .with_local_executor(executors.SynchronousLocalTaskExecutor())\n",
    "    .with_remote_executor(executors.MultiThreadingExecutor(max_tasks=5))\n",
    "    .build()\n",
    ")\n",
    "# this will now display the DAG and we'll see we have \"book ended\" the pipeline.\n",
    "parallel_dr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "52927cb8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>url</th>\n",
       "      <th>text</th>\n",
       "      <th>embedding</th>\n",
       "      <th>metadata</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>document_id</th>\n",
       "      <th>chunk_index</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"22\" valign=\"top\">c27f43ab-33b4-4a21-98c1-49a41b3b0c11</th>\n",
       "      <th>0</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Hamilton is a general-purpose framework to wri...</td>\n",
       "      <td>[-0.02026444487273693, 0.009682427160441875, 0...</td>\n",
       "      <td>{'Header 1': 'Welcome to Hamilton¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Directed Acyclic Graph (DAG) that can be execu...</td>\n",
       "      <td>[-0.03777306526899338, 0.0073875077068805695, ...</td>\n",
       "      <td>{'Header 1': 'Welcome to Hamilton¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>The ABC of Hamilton¶</td>\n",
       "      <td>[-0.01697148010134697, 0.02586626075208187, 0....</td>\n",
       "      <td>{'Header 1': 'Welcome to Hamilton¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Facilitate collaboration. By focusing on funct...</td>\n",
       "      <td>[-0.018855366855859756, -0.00249655288644135, ...</td>\n",
       "      <td>{'Header 1': 'Why should you use Hamilton?¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Visualizations can be generated directly from ...</td>\n",
       "      <td>[-0.047300878912210464, 0.031106166541576385, ...</td>\n",
       "      <td>{'Header 1': 'Why should you use Hamilton?¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Reduce development time. Hamilton dataflows ar...</td>\n",
       "      <td>[-0.013192436657845974, -0.025940468534827232,...</td>\n",
       "      <td>{'Header 1': 'Why should you use Hamilton?¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Hamilton Hub.</td>\n",
       "      <td>[-0.0641380250453949, -0.019515952095389366, 0...</td>\n",
       "      <td>{'Header 1': 'Why should you use Hamilton?¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Own your platform. Hamilton helps you integrat...</td>\n",
       "      <td>[-0.01695844531059265, -0.023515893146395683, ...</td>\n",
       "      <td>{'Header 1': 'Why should you use Hamilton?¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Scale your dataflow. Hamilton separates transf...</td>\n",
       "      <td>[0.0007336082635447383, 0.012326962314546108, ...</td>\n",
       "      <td>{'Header 1': 'Why should you use Hamilton?¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>enterprise data workloads.</td>\n",
       "      <td>[0.007231128867715597, 0.018477486446499825, 0...</td>\n",
       "      <td>{'Header 1': 'Why should you use Hamilton?¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Here’s a quick overview of benefits that Hamil...</td>\n",
       "      <td>[-0.04146569222211838, 0.01626516319811344, 0....</td>\n",
       "      <td>{'Header 1': 'Why should you use Hamilton?¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Can visualize lineage easily  \\n✅  \\n❌  \\n❌  \\...</td>\n",
       "      <td>[-0.03983769938349724, 0.01923440210521221, 0....</td>\n",
       "      <td>{'Header 1': 'Why should you use Hamilton?¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Runs anywhere python runs  \\n✅  \\n❌  \\n❌  \\n❌ ...</td>\n",
       "      <td>[-0.04583346098661423, 0.022130435332655907, 0...</td>\n",
       "      <td>{'Header 1': 'Why should you use Hamilton?¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>The following diagram gives a simplified overv...</td>\n",
       "      <td>[-0.04830301180481911, 0.021801836788654327, 0...</td>\n",
       "      <td>{'Header 1': 'Architecture Overview¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Functions &amp; Module. Transformations are regula...</td>\n",
       "      <td>[-0.041252683848142624, 0.04047299176454544, 0...</td>\n",
       "      <td>{'Header 1': 'Architecture Overview¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Driver &amp; FunctionGraph. The Driver will automa...</td>\n",
       "      <td>[-0.012794064357876778, 0.018729455769062042, ...</td>\n",
       "      <td>{'Header 1': 'Architecture Overview¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Visualization. The FunctionGraph can be visual...</td>\n",
       "      <td>[-0.045535918325185776, 0.034158188849687576, ...</td>\n",
       "      <td>{'Header 1': 'Architecture Overview¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Execution. When requesting variables, the Driv...</td>\n",
       "      <td>[0.0031941100023686886, 0.030165577307343483, ...</td>\n",
       "      <td>{'Header 1': 'Architecture Overview¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Multiple companies are doing cool stuff with H...</td>\n",
       "      <td>[-0.00601744232699275, 0.008543377742171288, 0...</td>\n",
       "      <td>{'Header 1': 'Who is using Hamilton?¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Transfix - Online featurization and prediction...</td>\n",
       "      <td>[-0.0015230675926432014, 0.010722102597355843,...</td>\n",
       "      <td>{'Header 1': 'Who is using Hamilton?¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>Habitat - Time-series feature engineering  \\nU...</td>\n",
       "      <td>[0.009830841794610023, 0.004250616300851107, 0...</td>\n",
       "      <td>{'Header 1': 'Who is using Hamilton?¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/</td>\n",
       "      <td>KI - Feature engineering</td>\n",
       "      <td>[-0.010076257400214672, 0.01848677173256874, 0...</td>\n",
       "      <td>{'Header 1': 'Who is using Hamilton?¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"27\" valign=\"top\">fea52d7e-5968-4f68-8c71-5b6bc7eef137</th>\n",
       "      <th>0</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>Hamilton comes with a fully open-source UI tha...</td>\n",
       "      <td>[-0.033495984971523285, 0.02976820431649685, 0...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>A feature/artifact catalog for browsing/connec...</td>\n",
       "      <td>[-0.029500646516680717, 0.027440618723630905, ...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>A project explorer for viewing curating projec...</td>\n",
       "      <td>[-0.034877486526966095, -0.015646569430828094,...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>If you’d like a video walkthrough on getting s...</td>\n",
       "      <td>[-0.04170287773013115, 0.01874636672437191, -0...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>The Hamilton UI is contained within a set of d...</td>\n",
       "      <td>[-0.06400912255048752, -0.009062161669135094, ...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>To run this, you’ll need:  \\nDocker installed ...</td>\n",
       "      <td>[-0.021686555817723274, 0.03889773041009903, 0...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>Start by ensuring that the hamilton repository...</td>\n",
       "      <td>[-0.04604955390095711, 0.027341920882463455, 0...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>./deployment/run.sh  \\nThis will do the follow...</td>\n",
       "      <td>[-0.014610188081860542, -0.008541340939700603,...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>This takes a bit of time! So be patient. The s...</td>\n",
       "      <td>[0.005524422507733107, -0.007484701462090015, ...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>If building the Docker containers from scratch...</td>\n",
       "      <td>[0.008969233371317387, 0.012474973686039448, 0...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td># from the hamilton/ui directory ./deployment/...</td>\n",
       "      <td>[-0.025990990921854973, 0.0099107064306736, 0....</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>Now that you have your server running, you can...</td>\n",
       "      <td>[-0.019784538075327873, 0.004920551087707281, ...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>Then, navigate to the project page (dashboard/...</td>\n",
       "      <td>[-0.012636549770832062, -0.014818817377090454,...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>from hamilton_sdk import adapters tracker = ad...</td>\n",
       "      <td>[0.0016246301820501685, -0.014947794377803802,...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>driver.Builder() .with_config(your_config) .wi...</td>\n",
       "      <td>[0.02870572917163372, 0.0004390304966364056, 0...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>Then run your DAG, and follow the links in the...</td>\n",
       "      <td>[-0.05947300046682358, 0.01847035251557827, -0...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td># we assume you're in the Hamilton repository ...</td>\n",
       "      <td>[-0.029487399384379387, -0.01613418199121952, ...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>--email &lt;email&gt; --project_id &lt;project_id&gt;</td>\n",
       "      <td>[0.01633821241557598, -0.01278590690344572, 0....</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>You should see links in the logs to the UI, wh...</td>\n",
       "      <td>[-0.03280032426118851, 0.024312302470207214, -...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>Once you get to the UI, you can navigate to th...</td>\n",
       "      <td>[-0.013666111044585705, -0.000534116174094379,...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>See below for a few screenshots of the UI.</td>\n",
       "      <td>[-0.08185029029846191, 0.0024983373004943132, ...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>The UI has the following features:</td>\n",
       "      <td>[-0.0673123225569725, 0.02698035165667534, 0.0...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Feat...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>Select DAG versions to compare + visualize.</td>\n",
       "      <td>[-0.02967393398284912, -0.04668116569519043, 0...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Feat...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>View functions, nodes, and assets across a his...</td>\n",
       "      <td>[-0.03586745634675026, 0.022120537236332893, 0...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Feat...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>View DAG shapes + code:</td>\n",
       "      <td>[-0.04795920476317406, -0.0044870213605463505,...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Feat...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>View a history of runs, telemetry on runs/comp...</td>\n",
       "      <td>[-0.014719597063958645, 0.01228130329400301, 0...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Feat...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>https://hamilton.dagworks.io/en/latest/concept...</td>\n",
       "      <td>Please reach out to us if you want to deploy o...</td>\n",
       "      <td>[0.013612204231321812, -0.003925923258066177, ...</td>\n",
       "      <td>{'Header 1': 'Hamilton UI¶', 'Header 2': 'Self...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                url  \\\n",
       "document_id                          chunk_index                                                      \n",
       "c27f43ab-33b4-4a21-98c1-49a41b3b0c11 0                      https://hamilton.dagworks.io/en/latest/   \n",
       "                                     1                      https://hamilton.dagworks.io/en/latest/   \n",
       "                                     2                      https://hamilton.dagworks.io/en/latest/   \n",
       "                                     3                      https://hamilton.dagworks.io/en/latest/   \n",
       "                                     4                      https://hamilton.dagworks.io/en/latest/   \n",
       "                                     5                      https://hamilton.dagworks.io/en/latest/   \n",
       "                                     6                      https://hamilton.dagworks.io/en/latest/   \n",
       "                                     7                      https://hamilton.dagworks.io/en/latest/   \n",
       "                                     8                      https://hamilton.dagworks.io/en/latest/   \n",
       "                                     9                      https://hamilton.dagworks.io/en/latest/   \n",
       "                                     10                     https://hamilton.dagworks.io/en/latest/   \n",
       "                                     11                     https://hamilton.dagworks.io/en/latest/   \n",
       "                                     12                     https://hamilton.dagworks.io/en/latest/   \n",
       "                                     13                     https://hamilton.dagworks.io/en/latest/   \n",
       "                                     14                     https://hamilton.dagworks.io/en/latest/   \n",
       "                                     15                     https://hamilton.dagworks.io/en/latest/   \n",
       "                                     16                     https://hamilton.dagworks.io/en/latest/   \n",
       "                                     17                     https://hamilton.dagworks.io/en/latest/   \n",
       "                                     18                     https://hamilton.dagworks.io/en/latest/   \n",
       "                                     19                     https://hamilton.dagworks.io/en/latest/   \n",
       "                                     20                     https://hamilton.dagworks.io/en/latest/   \n",
       "                                     21                     https://hamilton.dagworks.io/en/latest/   \n",
       "fea52d7e-5968-4f68-8c71-5b6bc7eef137 0            https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     1            https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     2            https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     3            https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     4            https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     5            https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     6            https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     7            https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     8            https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     9            https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     10           https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     11           https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     12           https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     13           https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     14           https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     15           https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     16           https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     17           https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     18           https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     19           https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     20           https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     21           https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     22           https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     23           https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     24           https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     25           https://hamilton.dagworks.io/en/latest/concept...   \n",
       "                                     26           https://hamilton.dagworks.io/en/latest/concept...   \n",
       "\n",
       "                                                                                               text  \\\n",
       "document_id                          chunk_index                                                      \n",
       "c27f43ab-33b4-4a21-98c1-49a41b3b0c11 0            Hamilton is a general-purpose framework to wri...   \n",
       "                                     1            Directed Acyclic Graph (DAG) that can be execu...   \n",
       "                                     2                                         The ABC of Hamilton¶   \n",
       "                                     3            Facilitate collaboration. By focusing on funct...   \n",
       "                                     4            Visualizations can be generated directly from ...   \n",
       "                                     5            Reduce development time. Hamilton dataflows ar...   \n",
       "                                     6                                                Hamilton Hub.   \n",
       "                                     7            Own your platform. Hamilton helps you integrat...   \n",
       "                                     8            Scale your dataflow. Hamilton separates transf...   \n",
       "                                     9                                   enterprise data workloads.   \n",
       "                                     10           Here’s a quick overview of benefits that Hamil...   \n",
       "                                     11           Can visualize lineage easily  \\n✅  \\n❌  \\n❌  \\...   \n",
       "                                     12           Runs anywhere python runs  \\n✅  \\n❌  \\n❌  \\n❌ ...   \n",
       "                                     13           The following diagram gives a simplified overv...   \n",
       "                                     14           Functions & Module. Transformations are regula...   \n",
       "                                     15           Driver & FunctionGraph. The Driver will automa...   \n",
       "                                     16           Visualization. The FunctionGraph can be visual...   \n",
       "                                     17           Execution. When requesting variables, the Driv...   \n",
       "                                     18           Multiple companies are doing cool stuff with H...   \n",
       "                                     19           Transfix - Online featurization and prediction...   \n",
       "                                     20           Habitat - Time-series feature engineering  \\nU...   \n",
       "                                     21                                    KI - Feature engineering   \n",
       "fea52d7e-5968-4f68-8c71-5b6bc7eef137 0            Hamilton comes with a fully open-source UI tha...   \n",
       "                                     1            A feature/artifact catalog for browsing/connec...   \n",
       "                                     2            A project explorer for viewing curating projec...   \n",
       "                                     3            If you’d like a video walkthrough on getting s...   \n",
       "                                     4            The Hamilton UI is contained within a set of d...   \n",
       "                                     5            To run this, you’ll need:  \\nDocker installed ...   \n",
       "                                     6            Start by ensuring that the hamilton repository...   \n",
       "                                     7            ./deployment/run.sh  \\nThis will do the follow...   \n",
       "                                     8            This takes a bit of time! So be patient. The s...   \n",
       "                                     9            If building the Docker containers from scratch...   \n",
       "                                     10           # from the hamilton/ui directory ./deployment/...   \n",
       "                                     11           Now that you have your server running, you can...   \n",
       "                                     12           Then, navigate to the project page (dashboard/...   \n",
       "                                     13           from hamilton_sdk import adapters tracker = ad...   \n",
       "                                     14           driver.Builder() .with_config(your_config) .wi...   \n",
       "                                     15           Then run your DAG, and follow the links in the...   \n",
       "                                     16           # we assume you're in the Hamilton repository ...   \n",
       "                                     17                   --email <email> --project_id <project_id>   \n",
       "                                     18           You should see links in the logs to the UI, wh...   \n",
       "                                     19           Once you get to the UI, you can navigate to th...   \n",
       "                                     20                  See below for a few screenshots of the UI.   \n",
       "                                     21                          The UI has the following features:   \n",
       "                                     22                 Select DAG versions to compare + visualize.   \n",
       "                                     23           View functions, nodes, and assets across a his...   \n",
       "                                     24                                     View DAG shapes + code:   \n",
       "                                     25           View a history of runs, telemetry on runs/comp...   \n",
       "                                     26           Please reach out to us if you want to deploy o...   \n",
       "\n",
       "                                                                                          embedding  \\\n",
       "document_id                          chunk_index                                                      \n",
       "c27f43ab-33b4-4a21-98c1-49a41b3b0c11 0            [-0.02026444487273693, 0.009682427160441875, 0...   \n",
       "                                     1            [-0.03777306526899338, 0.0073875077068805695, ...   \n",
       "                                     2            [-0.01697148010134697, 0.02586626075208187, 0....   \n",
       "                                     3            [-0.018855366855859756, -0.00249655288644135, ...   \n",
       "                                     4            [-0.047300878912210464, 0.031106166541576385, ...   \n",
       "                                     5            [-0.013192436657845974, -0.025940468534827232,...   \n",
       "                                     6            [-0.0641380250453949, -0.019515952095389366, 0...   \n",
       "                                     7            [-0.01695844531059265, -0.023515893146395683, ...   \n",
       "                                     8            [0.0007336082635447383, 0.012326962314546108, ...   \n",
       "                                     9            [0.007231128867715597, 0.018477486446499825, 0...   \n",
       "                                     10           [-0.04146569222211838, 0.01626516319811344, 0....   \n",
       "                                     11           [-0.03983769938349724, 0.01923440210521221, 0....   \n",
       "                                     12           [-0.04583346098661423, 0.022130435332655907, 0...   \n",
       "                                     13           [-0.04830301180481911, 0.021801836788654327, 0...   \n",
       "                                     14           [-0.041252683848142624, 0.04047299176454544, 0...   \n",
       "                                     15           [-0.012794064357876778, 0.018729455769062042, ...   \n",
       "                                     16           [-0.045535918325185776, 0.034158188849687576, ...   \n",
       "                                     17           [0.0031941100023686886, 0.030165577307343483, ...   \n",
       "                                     18           [-0.00601744232699275, 0.008543377742171288, 0...   \n",
       "                                     19           [-0.0015230675926432014, 0.010722102597355843,...   \n",
       "                                     20           [0.009830841794610023, 0.004250616300851107, 0...   \n",
       "                                     21           [-0.010076257400214672, 0.01848677173256874, 0...   \n",
       "fea52d7e-5968-4f68-8c71-5b6bc7eef137 0            [-0.033495984971523285, 0.02976820431649685, 0...   \n",
       "                                     1            [-0.029500646516680717, 0.027440618723630905, ...   \n",
       "                                     2            [-0.034877486526966095, -0.015646569430828094,...   \n",
       "                                     3            [-0.04170287773013115, 0.01874636672437191, -0...   \n",
       "                                     4            [-0.06400912255048752, -0.009062161669135094, ...   \n",
       "                                     5            [-0.021686555817723274, 0.03889773041009903, 0...   \n",
       "                                     6            [-0.04604955390095711, 0.027341920882463455, 0...   \n",
       "                                     7            [-0.014610188081860542, -0.008541340939700603,...   \n",
       "                                     8            [0.005524422507733107, -0.007484701462090015, ...   \n",
       "                                     9            [0.008969233371317387, 0.012474973686039448, 0...   \n",
       "                                     10           [-0.025990990921854973, 0.0099107064306736, 0....   \n",
       "                                     11           [-0.019784538075327873, 0.004920551087707281, ...   \n",
       "                                     12           [-0.012636549770832062, -0.014818817377090454,...   \n",
       "                                     13           [0.0016246301820501685, -0.014947794377803802,...   \n",
       "                                     14           [0.02870572917163372, 0.0004390304966364056, 0...   \n",
       "                                     15           [-0.05947300046682358, 0.01847035251557827, -0...   \n",
       "                                     16           [-0.029487399384379387, -0.01613418199121952, ...   \n",
       "                                     17           [0.01633821241557598, -0.01278590690344572, 0....   \n",
       "                                     18           [-0.03280032426118851, 0.024312302470207214, -...   \n",
       "                                     19           [-0.013666111044585705, -0.000534116174094379,...   \n",
       "                                     20           [-0.08185029029846191, 0.0024983373004943132, ...   \n",
       "                                     21           [-0.0673123225569725, 0.02698035165667534, 0.0...   \n",
       "                                     22           [-0.02967393398284912, -0.04668116569519043, 0...   \n",
       "                                     23           [-0.03586745634675026, 0.022120537236332893, 0...   \n",
       "                                     24           [-0.04795920476317406, -0.0044870213605463505,...   \n",
       "                                     25           [-0.014719597063958645, 0.01228130329400301, 0...   \n",
       "                                     26           [0.013612204231321812, -0.003925923258066177, ...   \n",
       "\n",
       "                                                                                           metadata  \n",
       "document_id                          chunk_index                                                     \n",
       "c27f43ab-33b4-4a21-98c1-49a41b3b0c11 0                         {'Header 1': 'Welcome to Hamilton¶'}  \n",
       "                                     1                         {'Header 1': 'Welcome to Hamilton¶'}  \n",
       "                                     2                         {'Header 1': 'Welcome to Hamilton¶'}  \n",
       "                                     3                {'Header 1': 'Why should you use Hamilton?¶'}  \n",
       "                                     4                {'Header 1': 'Why should you use Hamilton?¶'}  \n",
       "                                     5                {'Header 1': 'Why should you use Hamilton?¶'}  \n",
       "                                     6                {'Header 1': 'Why should you use Hamilton?¶'}  \n",
       "                                     7                {'Header 1': 'Why should you use Hamilton?¶'}  \n",
       "                                     8                {'Header 1': 'Why should you use Hamilton?¶'}  \n",
       "                                     9                {'Header 1': 'Why should you use Hamilton?¶'}  \n",
       "                                     10               {'Header 1': 'Why should you use Hamilton?¶'}  \n",
       "                                     11               {'Header 1': 'Why should you use Hamilton?¶'}  \n",
       "                                     12               {'Header 1': 'Why should you use Hamilton?¶'}  \n",
       "                                     13                      {'Header 1': 'Architecture Overview¶'}  \n",
       "                                     14                      {'Header 1': 'Architecture Overview¶'}  \n",
       "                                     15                      {'Header 1': 'Architecture Overview¶'}  \n",
       "                                     16                      {'Header 1': 'Architecture Overview¶'}  \n",
       "                                     17                      {'Header 1': 'Architecture Overview¶'}  \n",
       "                                     18                     {'Header 1': 'Who is using Hamilton?¶'}  \n",
       "                                     19                     {'Header 1': 'Who is using Hamilton?¶'}  \n",
       "                                     20                     {'Header 1': 'Who is using Hamilton?¶'}  \n",
       "                                     21                     {'Header 1': 'Who is using Hamilton?¶'}  \n",
       "fea52d7e-5968-4f68-8c71-5b6bc7eef137 0                                 {'Header 1': 'Hamilton UI¶'}  \n",
       "                                     1                                 {'Header 1': 'Hamilton UI¶'}  \n",
       "                                     2                                 {'Header 1': 'Hamilton UI¶'}  \n",
       "                                     3            {'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...  \n",
       "                                     4            {'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...  \n",
       "                                     5            {'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...  \n",
       "                                     6            {'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...  \n",
       "                                     7            {'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...  \n",
       "                                     8            {'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...  \n",
       "                                     9            {'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...  \n",
       "                                     10           {'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...  \n",
       "                                     11           {'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...  \n",
       "                                     12           {'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...  \n",
       "                                     13           {'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...  \n",
       "                                     14           {'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...  \n",
       "                                     15           {'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...  \n",
       "                                     16           {'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...  \n",
       "                                     17           {'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...  \n",
       "                                     18           {'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...  \n",
       "                                     19           {'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...  \n",
       "                                     20           {'Header 1': 'Hamilton UI¶', 'Header 2': 'Over...  \n",
       "                                     21           {'Header 1': 'Hamilton UI¶', 'Header 2': 'Feat...  \n",
       "                                     22           {'Header 1': 'Hamilton UI¶', 'Header 2': 'Feat...  \n",
       "                                     23           {'Header 1': 'Hamilton UI¶', 'Header 2': 'Feat...  \n",
       "                                     24           {'Header 1': 'Hamilton UI¶', 'Header 2': 'Feat...  \n",
       "                                     25           {'Header 1': 'Hamilton UI¶', 'Header 2': 'Feat...  \n",
       "                                     26           {'Header 1': 'Hamilton UI¶', 'Header 2': 'Self...  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's execute it\n",
    "urls = [\"https://hamilton.apache.org/\", \"https://hamilton.apache.org/concepts/ui\"] # some list of URLs\n",
    "result = parallel_dr.execute([\"collected_dataframes\"], inputs={\"urls\":urls})\n",
    "result[\"collected_dataframes\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ddd198a",
   "metadata": {},
   "source": [
    "### Options 3: Use PySpark, Dask, or Ray\n",
    "\n",
    "We skip showing that here - but [here's a blog post](https://blog.dagworks.io/p/rag-ingestion-and-chunking-using) on how with minimal changes you could get the pipeline we just wrote to processes a lot more documents easily. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eda7c980",
   "metadata": {},
   "source": [
    "## 2. How did the Hamilton driver know what our pipeline was?\n",
    "By using cell magic, we have incrementally been creating the `doc_pipeline` python module.\n",
    "\n",
    "Then when we did `driver.Builder().with_modules(doc_pipeline).build()` Hamilton took that python module and created the pipeline from it to execute. You can use the `%print_module` line magic to see the state of the python module at any time. You can then also use this as a means to export your notebook code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "63595961",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-03T22:36:28.862839Z",
     "start_time": "2024-05-03T22:36:28.783527Z"
    },
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>pre { line-height: 125%; }\n",
       "td.linenos .normal { color: inherit; background-color: transparent; padding-left: 5px; padding-right: 5px; }\n",
       "span.linenos { color: inherit; background-color: transparent; padding-left: 5px; padding-right: 5px; }\n",
       "td.linenos .special { color: #000000; background-color: #ffffc0; padding-left: 5px; padding-right: 5px; }\n",
       "span.linenos.special { color: #000000; background-color: #ffffc0; padding-left: 5px; padding-right: 5px; }\n",
       ".output_html .hll { background-color: #ffffcc }\n",
       ".output_html { background: #f8f8f8; }\n",
       ".output_html .c { color: #3D7B7B; font-style: italic } /* Comment */\n",
       ".output_html .err { border: 1px solid #FF0000 } /* Error */\n",
       ".output_html .k { color: #008000; font-weight: bold } /* Keyword */\n",
       ".output_html .o { color: #666666 } /* Operator */\n",
       ".output_html .ch { color: #3D7B7B; font-style: italic } /* Comment.Hashbang */\n",
       ".output_html .cm { color: #3D7B7B; font-style: italic } /* Comment.Multiline */\n",
       ".output_html .cp { color: #9C6500 } /* Comment.Preproc */\n",
       ".output_html .cpf { color: #3D7B7B; font-style: italic } /* Comment.PreprocFile */\n",
       ".output_html .c1 { color: #3D7B7B; font-style: italic } /* Comment.Single */\n",
       ".output_html .cs { color: #3D7B7B; font-style: italic } /* Comment.Special */\n",
       ".output_html .gd { color: #A00000 } /* Generic.Deleted */\n",
       ".output_html .ge { font-style: italic } /* Generic.Emph */\n",
       ".output_html .ges { font-weight: bold; font-style: italic } /* Generic.EmphStrong */\n",
       ".output_html .gr { color: #E40000 } /* Generic.Error */\n",
       ".output_html .gh { color: #000080; font-weight: bold } /* Generic.Heading */\n",
       ".output_html .gi { color: #008400 } /* Generic.Inserted */\n",
       ".output_html .go { color: #717171 } /* Generic.Output */\n",
       ".output_html .gp { color: #000080; font-weight: bold } /* Generic.Prompt */\n",
       ".output_html .gs { font-weight: bold } /* Generic.Strong */\n",
       ".output_html .gu { color: #800080; font-weight: bold } /* Generic.Subheading */\n",
       ".output_html .gt { color: #0044DD } /* Generic.Traceback */\n",
       ".output_html .kc { color: #008000; font-weight: bold } /* Keyword.Constant */\n",
       ".output_html .kd { color: #008000; font-weight: bold } /* Keyword.Declaration */\n",
       ".output_html .kn { color: #008000; font-weight: bold } /* Keyword.Namespace */\n",
       ".output_html .kp { color: #008000 } /* Keyword.Pseudo */\n",
       ".output_html .kr { color: #008000; font-weight: bold } /* Keyword.Reserved */\n",
       ".output_html .kt { color: #B00040 } /* Keyword.Type */\n",
       ".output_html .m { color: #666666 } /* Literal.Number */\n",
       ".output_html .s { color: #BA2121 } /* Literal.String */\n",
       ".output_html .na { color: #687822 } /* Name.Attribute */\n",
       ".output_html .nb { color: #008000 } /* Name.Builtin */\n",
       ".output_html .nc { color: #0000FF; font-weight: bold } /* Name.Class */\n",
       ".output_html .no { color: #880000 } /* Name.Constant */\n",
       ".output_html .nd { color: #AA22FF } /* Name.Decorator */\n",
       ".output_html .ni { color: #717171; font-weight: bold } /* Name.Entity */\n",
       ".output_html .ne { color: #CB3F38; font-weight: bold } /* Name.Exception */\n",
       ".output_html .nf { color: #0000FF } /* Name.Function */\n",
       ".output_html .nl { color: #767600 } /* Name.Label */\n",
       ".output_html .nn { color: #0000FF; font-weight: bold } /* Name.Namespace */\n",
       ".output_html .nt { color: #008000; font-weight: bold } /* Name.Tag */\n",
       ".output_html .nv { color: #19177C } /* Name.Variable */\n",
       ".output_html .ow { color: #AA22FF; font-weight: bold } /* Operator.Word */\n",
       ".output_html .w { color: #bbbbbb } /* Text.Whitespace */\n",
       ".output_html .mb { color: #666666 } /* Literal.Number.Bin */\n",
       ".output_html .mf { color: #666666 } /* Literal.Number.Float */\n",
       ".output_html .mh { color: #666666 } /* Literal.Number.Hex */\n",
       ".output_html .mi { color: #666666 } /* Literal.Number.Integer */\n",
       ".output_html .mo { color: #666666 } /* Literal.Number.Oct */\n",
       ".output_html .sa { color: #BA2121 } /* Literal.String.Affix */\n",
       ".output_html .sb { color: #BA2121 } /* Literal.String.Backtick */\n",
       ".output_html .sc { color: #BA2121 } /* Literal.String.Char */\n",
       ".output_html .dl { color: #BA2121 } /* Literal.String.Delimiter */\n",
       ".output_html .sd { color: #BA2121; font-style: italic } /* Literal.String.Doc */\n",
       ".output_html .s2 { color: #BA2121 } /* Literal.String.Double */\n",
       ".output_html .se { color: #AA5D1F; font-weight: bold } /* Literal.String.Escape */\n",
       ".output_html .sh { color: #BA2121 } /* Literal.String.Heredoc */\n",
       ".output_html .si { color: #A45A77; font-weight: bold } /* Literal.String.Interpol */\n",
       ".output_html .sx { color: #008000 } /* Literal.String.Other */\n",
       ".output_html .sr { color: #A45A77 } /* Literal.String.Regex */\n",
       ".output_html .s1 { color: #BA2121 } /* Literal.String.Single */\n",
       ".output_html .ss { color: #19177C } /* Literal.String.Symbol */\n",
       ".output_html .bp { color: #008000 } /* Name.Builtin.Pseudo */\n",
       ".output_html .fm { color: #0000FF } /* Name.Function.Magic */\n",
       ".output_html .vc { color: #19177C } /* Name.Variable.Class */\n",
       ".output_html .vg { color: #19177C } /* Name.Variable.Global */\n",
       ".output_html .vi { color: #19177C } /* Name.Variable.Instance */\n",
       ".output_html .vm { color: #19177C } /* Name.Variable.Magic */\n",
       ".output_html .il { color: #666666 } /* Literal.Number.Integer.Long */</style><div class=\"highlight\"><pre><span></span><span class=\"kn\">from</span> <span class=\"nn\">typing</span> <span class=\"kn\">import</span> <span class=\"n\">NamedTuple</span><span class=\"p\">,</span> <span class=\"n\">Optional</span>\n",
       "\n",
       "<span class=\"k\">class</span> <span class=\"nc\">Chunk</span><span class=\"p\">(</span><span class=\"n\">NamedTuple</span><span class=\"p\">):</span>\n",
       "<span class=\"w\">    </span><span class=\"sd\">&quot;&quot;&quot;Chunk of a Document - text &amp; embedding.&quot;&quot;&quot;</span>\n",
       "    <span class=\"n\">index</span><span class=\"p\">:</span> <span class=\"nb\">int</span>\n",
       "    <span class=\"n\">document_id</span><span class=\"p\">:</span> <span class=\"nb\">str</span>\n",
       "    <span class=\"n\">text</span><span class=\"p\">:</span> <span class=\"nb\">str</span>\n",
       "    <span class=\"n\">embedding</span><span class=\"p\">:</span> <span class=\"n\">Optional</span><span class=\"p\">[</span><span class=\"nb\">list</span><span class=\"p\">[</span><span class=\"nb\">float</span><span class=\"p\">]]</span>\n",
       "    <span class=\"n\">metadata</span><span class=\"p\">:</span> <span class=\"n\">Optional</span><span class=\"p\">[</span><span class=\"nb\">dict</span><span class=\"p\">[</span><span class=\"nb\">str</span><span class=\"p\">,</span> <span class=\"nb\">str</span><span class=\"p\">]]</span>\n",
       "    \n",
       "    <span class=\"k\">def</span> <span class=\"nf\">add_embedding</span><span class=\"p\">(</span><span class=\"bp\">self</span><span class=\"p\">,</span> <span class=\"n\">embedding</span><span class=\"p\">:</span> <span class=\"nb\">list</span><span class=\"p\">[</span><span class=\"nb\">float</span><span class=\"p\">])</span> <span class=\"o\">-&gt;</span> <span class=\"s2\">&quot;Chunk&quot;</span><span class=\"p\">:</span>\n",
       "<span class=\"w\">        </span><span class=\"sd\">&quot;&quot;&quot;Required to update chunk with embeddings&quot;&quot;&quot;</span>\n",
       "        <span class=\"k\">return</span> <span class=\"n\">Chunk</span><span class=\"p\">(</span><span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">index</span><span class=\"p\">,</span> <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">document_id</span><span class=\"p\">,</span> <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">text</span><span class=\"p\">,</span> <span class=\"n\">embedding</span><span class=\"p\">,</span> <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">metadata</span><span class=\"p\">)</span>\n",
       "    \n",
       "    <span class=\"k\">def</span> <span class=\"nf\">add_metadata</span><span class=\"p\">(</span><span class=\"bp\">self</span><span class=\"p\">,</span> <span class=\"n\">metadata</span><span class=\"p\">:</span> <span class=\"nb\">dict</span><span class=\"p\">[</span><span class=\"nb\">str</span><span class=\"p\">,</span> <span class=\"nb\">str</span><span class=\"p\">])</span> <span class=\"o\">-&gt;</span> <span class=\"s2\">&quot;Chunk&quot;</span><span class=\"p\">:</span>\n",
       "<span class=\"w\">        </span><span class=\"sd\">&quot;&quot;&quot;Required to update chunk with metadata&quot;&quot;&quot;</span>\n",
       "        <span class=\"k\">return</span> <span class=\"n\">Chunk</span><span class=\"p\">(</span><span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">index</span><span class=\"p\">,</span> <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">document_id</span><span class=\"p\">,</span> <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">text</span><span class=\"p\">,</span> <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">embedding</span><span class=\"p\">,</span> <span class=\"n\">metadata</span><span class=\"p\">)</span>\n",
       "\n",
       "<span class=\"k\">class</span> <span class=\"nc\">Document</span><span class=\"p\">(</span><span class=\"n\">NamedTuple</span><span class=\"p\">):</span>\n",
       "<span class=\"w\">    </span><span class=\"sd\">&quot;&quot;&quot;Document containing a full raw text, along with pointers to chunks.&quot;&quot;&quot;</span>\n",
       "    <span class=\"nb\">id</span><span class=\"p\">:</span> <span class=\"nb\">str</span>\n",
       "    <span class=\"n\">url</span><span class=\"p\">:</span> <span class=\"nb\">str</span>\n",
       "    <span class=\"n\">raw_text</span><span class=\"p\">:</span> <span class=\"nb\">str</span>\n",
       "    <span class=\"n\">chunks</span><span class=\"p\">:</span> <span class=\"n\">Optional</span><span class=\"p\">[</span><span class=\"nb\">list</span><span class=\"p\">[</span><span class=\"n\">Chunk</span><span class=\"p\">]]</span>\n",
       "    \n",
       "    <span class=\"k\">def</span> <span class=\"nf\">add_chunks</span><span class=\"p\">(</span><span class=\"bp\">self</span><span class=\"p\">,</span> <span class=\"n\">chunks</span><span class=\"p\">:</span> <span class=\"nb\">list</span><span class=\"p\">[</span><span class=\"n\">Chunk</span><span class=\"p\">])</span> <span class=\"o\">-&gt;</span> <span class=\"s2\">&quot;Document&quot;</span><span class=\"p\">:</span>\n",
       "<span class=\"w\">        </span><span class=\"sd\">&quot;&quot;&quot;Required to update the document when Chunks are created&quot;&quot;&quot;</span>\n",
       "        <span class=\"k\">return</span> <span class=\"n\">Document</span><span class=\"p\">(</span><span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">id</span><span class=\"p\">,</span> <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">url</span><span class=\"p\">,</span> <span class=\"bp\">self</span><span class=\"o\">.</span><span class=\"n\">raw_text</span><span class=\"p\">,</span> <span class=\"n\">chunks</span><span class=\"p\">)</span>\n",
       "\n",
       "\n",
       "<span class=\"kn\">import</span> <span class=\"nn\">requests</span> \n",
       "<span class=\"kn\">import</span> <span class=\"nn\">re</span>\n",
       "<span class=\"kn\">import</span> <span class=\"nn\">uuid</span>\n",
       "\n",
       "<span class=\"k\">def</span> <span class=\"nf\">html_regex</span><span class=\"p\">()</span> <span class=\"o\">-&gt;</span> <span class=\"nb\">str</span><span class=\"p\">:</span>\n",
       "<span class=\"w\">    </span><span class=\"sd\">&quot;&quot;&quot;Context dependent logic for getting the right part of the HTML document.&quot;&quot;&quot;</span>\n",
       "    <span class=\"k\">return</span> <span class=\"sa\">r</span><span class=\"s1\">&#39;&lt;article role=&quot;main&quot; id=&quot;furo-main-content&quot;&gt;(.*?)&lt;/article&gt;&#39;</span>\n",
       "\n",
       "<span class=\"k\">def</span> <span class=\"nf\">raw_document</span><span class=\"p\">(</span><span class=\"n\">url</span><span class=\"p\">:</span> <span class=\"nb\">str</span><span class=\"p\">,</span> <span class=\"n\">html_regex</span><span class=\"p\">:</span> <span class=\"nb\">str</span><span class=\"p\">)</span> <span class=\"o\">-&gt;</span> <span class=\"n\">Document</span><span class=\"p\">:</span>\n",
       "<span class=\"w\">    </span><span class=\"sd\">&quot;&quot;&quot;Loads and parses the HTML from a URL, returning the html text of interest.</span>\n",
       "\n",
       "<span class=\"sd\">    :param url: the url to pull.</span>\n",
       "<span class=\"sd\">    :param html_regex: the regext to use to get the contents out of.</span>\n",
       "<span class=\"sd\">    :return: sub-portion of the HTML</span>\n",
       "<span class=\"sd\">    &quot;&quot;&quot;</span>\n",
       "    <span class=\"n\">html_text</span> <span class=\"o\">=</span> <span class=\"n\">requests</span><span class=\"o\">.</span><span class=\"n\">get</span><span class=\"p\">(</span><span class=\"n\">url</span><span class=\"p\">)</span><span class=\"o\">.</span><span class=\"n\">text</span>\n",
       "    <span class=\"n\">article</span> <span class=\"o\">=</span> <span class=\"n\">re</span><span class=\"o\">.</span><span class=\"n\">findall</span><span class=\"p\">(</span><span class=\"n\">html_regex</span><span class=\"p\">,</span> <span class=\"n\">html_text</span><span class=\"p\">,</span> <span class=\"n\">re</span><span class=\"o\">.</span><span class=\"n\">DOTALL</span><span class=\"p\">)</span>\n",
       "    <span class=\"k\">if</span> <span class=\"ow\">not</span> <span class=\"n\">article</span><span class=\"p\">:</span>\n",
       "        <span class=\"k\">raise</span> <span class=\"ne\">ValueError</span><span class=\"p\">(</span><span class=\"sa\">f</span><span class=\"s2\">&quot;No article found in </span><span class=\"si\">{</span><span class=\"n\">url</span><span class=\"si\">}</span><span class=\"s2\">&quot;</span><span class=\"p\">)</span>\n",
       "    <span class=\"n\">raw_text</span> <span class=\"o\">=</span> <span class=\"n\">article</span><span class=\"p\">[</span><span class=\"mi\">0</span><span class=\"p\">]</span><span class=\"o\">.</span><span class=\"n\">strip</span><span class=\"p\">()</span>\n",
       "    <span class=\"k\">return</span> <span class=\"n\">Document</span><span class=\"p\">(</span><span class=\"nb\">str</span><span class=\"p\">(</span><span class=\"n\">uuid</span><span class=\"o\">.</span><span class=\"n\">uuid4</span><span class=\"p\">()),</span> <span class=\"n\">url</span><span class=\"p\">,</span> <span class=\"n\">raw_text</span><span class=\"p\">,</span> <span class=\"kc\">None</span><span class=\"p\">)</span>\n",
       "\n",
       "\n",
       "\n",
       "<span class=\"kn\">from</span> <span class=\"nn\">langchain</span> <span class=\"kn\">import</span> <span class=\"n\">text_splitter</span>\n",
       "\n",
       "<span class=\"k\">def</span> <span class=\"nf\">html_chunker</span><span class=\"p\">()</span> <span class=\"o\">-&gt;</span> <span class=\"n\">text_splitter</span><span class=\"o\">.</span><span class=\"n\">HTMLHeaderTextSplitter</span><span class=\"p\">:</span>\n",
       "<span class=\"w\">    </span><span class=\"sd\">&quot;&quot;&quot;Object to help split HTML into chunks&quot;&quot;&quot;</span>\n",
       "    <span class=\"n\">headers_to_split_on</span> <span class=\"o\">=</span> <span class=\"p\">[</span>\n",
       "        <span class=\"p\">(</span><span class=\"s2\">&quot;h1&quot;</span><span class=\"p\">,</span> <span class=\"s2\">&quot;Header 1&quot;</span><span class=\"p\">),</span>\n",
       "        <span class=\"p\">(</span><span class=\"s2\">&quot;h2&quot;</span><span class=\"p\">,</span> <span class=\"s2\">&quot;Header 2&quot;</span><span class=\"p\">),</span>\n",
       "        <span class=\"p\">(</span><span class=\"s2\">&quot;h3&quot;</span><span class=\"p\">,</span> <span class=\"s2\">&quot;Header 3&quot;</span><span class=\"p\">),</span>\n",
       "    <span class=\"p\">]</span>\n",
       "    <span class=\"n\">html_chunker</span> <span class=\"o\">=</span> <span class=\"n\">text_splitter</span><span class=\"o\">.</span><span class=\"n\">HTMLHeaderTextSplitter</span><span class=\"p\">(</span>\n",
       "        <span class=\"n\">headers_to_split_on</span><span class=\"o\">=</span><span class=\"n\">headers_to_split_on</span>\n",
       "    <span class=\"p\">)</span>\n",
       "    <span class=\"k\">return</span> <span class=\"n\">html_chunker</span>\n",
       "\n",
       "<span class=\"k\">def</span> <span class=\"nf\">text_chunker</span><span class=\"p\">(</span>\n",
       "        <span class=\"n\">chunk_size</span><span class=\"p\">:</span> <span class=\"nb\">int</span> <span class=\"o\">=</span> <span class=\"mi\">256</span><span class=\"p\">,</span> \n",
       "        <span class=\"n\">chunk_overlap</span><span class=\"p\">:</span> <span class=\"nb\">int</span> <span class=\"o\">=</span> <span class=\"mi\">0</span><span class=\"p\">)</span> <span class=\"o\">-&gt;</span> <span class=\"n\">text_splitter</span><span class=\"o\">.</span><span class=\"n\">RecursiveCharacterTextSplitter</span><span class=\"p\">:</span>\n",
       "<span class=\"w\">    </span><span class=\"sd\">&quot;&quot;&quot;Object to further split chunks&quot;&quot;&quot;</span>\n",
       "    <span class=\"k\">return</span> <span class=\"n\">text_splitter</span><span class=\"o\">.</span><span class=\"n\">RecursiveCharacterTextSplitter</span><span class=\"p\">(</span>\n",
       "        <span class=\"n\">chunk_size</span><span class=\"o\">=</span><span class=\"n\">chunk_size</span><span class=\"p\">,</span> <span class=\"n\">chunk_overlap</span><span class=\"o\">=</span><span class=\"n\">chunk_overlap</span>\n",
       "    <span class=\"p\">)</span>\n",
       "\n",
       "<span class=\"k\">def</span> <span class=\"nf\">chunked_document</span><span class=\"p\">(</span>\n",
       "    <span class=\"n\">raw_document</span><span class=\"p\">:</span> <span class=\"n\">Document</span><span class=\"p\">,</span>\n",
       "    <span class=\"n\">html_chunker</span><span class=\"p\">:</span> <span class=\"n\">text_splitter</span><span class=\"o\">.</span><span class=\"n\">HTMLHeaderTextSplitter</span><span class=\"p\">,</span>\n",
       "    <span class=\"n\">text_chunker</span><span class=\"p\">:</span> <span class=\"n\">text_splitter</span><span class=\"o\">.</span><span class=\"n\">RecursiveCharacterTextSplitter</span><span class=\"p\">,</span>\n",
       "<span class=\"p\">)</span> <span class=\"o\">-&gt;</span> <span class=\"n\">Document</span><span class=\"p\">:</span>\n",
       "<span class=\"w\">    </span><span class=\"sd\">&quot;&quot;&quot;This function takes in HTML, chunks the HTML, and then chunks it into text chunks.&quot;&quot;&quot;</span>\n",
       "    <span class=\"n\">header_splits</span> <span class=\"o\">=</span> <span class=\"n\">html_chunker</span><span class=\"o\">.</span><span class=\"n\">split_text</span><span class=\"p\">(</span><span class=\"n\">raw_document</span><span class=\"o\">.</span><span class=\"n\">raw_text</span><span class=\"p\">)</span>\n",
       "    <span class=\"n\">text_chunks</span> <span class=\"o\">=</span> <span class=\"n\">text_chunker</span><span class=\"o\">.</span><span class=\"n\">split_documents</span><span class=\"p\">(</span><span class=\"n\">header_splits</span><span class=\"p\">)</span>\n",
       "    <span class=\"n\">chunks</span> <span class=\"o\">=</span> <span class=\"p\">[]</span>\n",
       "    <span class=\"k\">for</span> <span class=\"n\">i</span><span class=\"p\">,</span> <span class=\"n\">text_chunk</span> <span class=\"ow\">in</span> <span class=\"nb\">enumerate</span><span class=\"p\">(</span><span class=\"n\">text_chunks</span><span class=\"p\">):</span>\n",
       "        <span class=\"n\">chunks</span><span class=\"o\">.</span><span class=\"n\">append</span><span class=\"p\">(</span>\n",
       "            <span class=\"n\">Chunk</span><span class=\"p\">(</span>\n",
       "                <span class=\"n\">index</span><span class=\"o\">=</span><span class=\"n\">i</span><span class=\"p\">,</span>\n",
       "                <span class=\"n\">document_id</span><span class=\"o\">=</span><span class=\"n\">raw_document</span><span class=\"o\">.</span><span class=\"n\">id</span><span class=\"p\">,</span>\n",
       "                <span class=\"n\">text</span><span class=\"o\">=</span><span class=\"n\">text_chunk</span><span class=\"o\">.</span><span class=\"n\">page_content</span><span class=\"p\">,</span>\n",
       "                <span class=\"n\">embedding</span><span class=\"o\">=</span><span class=\"kc\">None</span><span class=\"p\">,</span>\n",
       "                <span class=\"n\">metadata</span><span class=\"o\">=</span><span class=\"n\">text_chunk</span><span class=\"o\">.</span><span class=\"n\">metadata</span><span class=\"p\">,</span>\n",
       "            <span class=\"p\">)</span>\n",
       "        <span class=\"p\">)</span>\n",
       "    <span class=\"c1\"># create new raw_document object</span>\n",
       "    <span class=\"n\">raw_document</span> <span class=\"o\">=</span> <span class=\"n\">raw_document</span><span class=\"o\">.</span><span class=\"n\">add_chunks</span><span class=\"p\">(</span><span class=\"n\">chunks</span><span class=\"p\">)</span>\n",
       "    <span class=\"k\">return</span> <span class=\"n\">raw_document</span>\n",
       "\n",
       "\n",
       "<span class=\"kn\">import</span> <span class=\"nn\">openai</span>\n",
       "\n",
       "<span class=\"k\">def</span> <span class=\"nf\">client</span><span class=\"p\">()</span> <span class=\"o\">-&gt;</span> <span class=\"n\">openai</span><span class=\"o\">.</span><span class=\"n\">OpenAI</span><span class=\"p\">:</span>\n",
       "    <span class=\"k\">return</span> <span class=\"n\">openai</span><span class=\"o\">.</span><span class=\"n\">OpenAI</span><span class=\"p\">()</span>\n",
       "\n",
       "<span class=\"k\">def</span> <span class=\"nf\">embedded_document</span><span class=\"p\">(</span>\n",
       "    <span class=\"n\">chunked_document</span><span class=\"p\">:</span> <span class=\"n\">Document</span><span class=\"p\">,</span>\n",
       "    <span class=\"n\">client</span><span class=\"p\">:</span> <span class=\"n\">openai</span><span class=\"o\">.</span><span class=\"n\">OpenAI</span><span class=\"p\">,</span>\n",
       "<span class=\"p\">)</span> <span class=\"o\">-&gt;</span> <span class=\"n\">Document</span><span class=\"p\">:</span>\n",
       "<span class=\"w\">    </span><span class=\"sd\">&quot;&quot;&quot;This function takes in a list of documents and outputs a list of documents with embeddings.</span>\n",
       "<span class=\"sd\">    &quot;&quot;&quot;</span>\n",
       "    <span class=\"k\">for</span> <span class=\"n\">idx</span><span class=\"p\">,</span> <span class=\"n\">chunk</span> <span class=\"ow\">in</span> <span class=\"nb\">enumerate</span><span class=\"p\">(</span><span class=\"n\">chunked_document</span><span class=\"o\">.</span><span class=\"n\">chunks</span><span class=\"p\">):</span> \n",
       "        <span class=\"n\">response</span> <span class=\"o\">=</span> <span class=\"n\">client</span><span class=\"o\">.</span><span class=\"n\">embeddings</span><span class=\"o\">.</span><span class=\"n\">create</span><span class=\"p\">(</span><span class=\"nb\">input</span><span class=\"o\">=</span><span class=\"n\">chunk</span><span class=\"o\">.</span><span class=\"n\">text</span><span class=\"p\">,</span> \n",
       "                                            <span class=\"n\">model</span><span class=\"o\">=</span><span class=\"s2\">&quot;text-embedding-3-small&quot;</span><span class=\"p\">)</span>\n",
       "        <span class=\"n\">chunk</span> <span class=\"o\">=</span> <span class=\"n\">chunk</span><span class=\"o\">.</span><span class=\"n\">add_embedding</span><span class=\"p\">(</span><span class=\"n\">response</span><span class=\"o\">.</span><span class=\"n\">data</span><span class=\"p\">[</span><span class=\"mi\">0</span><span class=\"p\">]</span><span class=\"o\">.</span><span class=\"n\">embedding</span><span class=\"p\">)</span>\n",
       "        <span class=\"c1\"># mutate existing document </span>\n",
       "        <span class=\"n\">chunked_document</span><span class=\"o\">.</span><span class=\"n\">chunks</span><span class=\"p\">[</span><span class=\"n\">idx</span><span class=\"p\">]</span> <span class=\"o\">=</span> <span class=\"n\">chunk</span>\n",
       "    <span class=\"k\">return</span> <span class=\"n\">chunked_document</span>\n",
       "\n",
       "\n",
       "<span class=\"kn\">import</span> <span class=\"nn\">pandas</span> <span class=\"k\">as</span> <span class=\"nn\">pd</span>\n",
       "\n",
       "<span class=\"k\">def</span> <span class=\"nf\">store</span><span class=\"p\">(</span>\n",
       "    <span class=\"n\">embedded_document</span><span class=\"p\">:</span> <span class=\"n\">Document</span><span class=\"p\">,</span>\n",
       "<span class=\"p\">)</span> <span class=\"o\">-&gt;</span> <span class=\"n\">pd</span><span class=\"o\">.</span><span class=\"n\">DataFrame</span><span class=\"p\">:</span>\n",
       "<span class=\"w\">    </span><span class=\"sd\">&quot;&quot;&quot;Function to index &amp; store the document. Here we just put it into a pandas dataframe.&quot;&quot;&quot;</span>\n",
       "    <span class=\"c1\"># make a pandas dataframe from the document</span>\n",
       "    <span class=\"c1\"># want a column for the document id, the chunk index, the text, the embedding, and the metadata</span>\n",
       "    <span class=\"n\">rows</span> <span class=\"o\">=</span> <span class=\"p\">[]</span>\n",
       "    <span class=\"k\">for</span> <span class=\"n\">chunk</span> <span class=\"ow\">in</span> <span class=\"n\">embedded_document</span><span class=\"o\">.</span><span class=\"n\">chunks</span><span class=\"p\">:</span>\n",
       "        <span class=\"c1\"># need to </span>\n",
       "        <span class=\"n\">rows</span><span class=\"o\">.</span><span class=\"n\">append</span><span class=\"p\">(</span>\n",
       "            <span class=\"p\">{</span>\n",
       "                <span class=\"s2\">&quot;document_id&quot;</span><span class=\"p\">:</span> <span class=\"n\">embedded_document</span><span class=\"o\">.</span><span class=\"n\">id</span><span class=\"p\">,</span>\n",
       "                <span class=\"s2\">&quot;chunk_index&quot;</span><span class=\"p\">:</span> <span class=\"n\">chunk</span><span class=\"o\">.</span><span class=\"n\">index</span><span class=\"p\">,</span>\n",
       "                <span class=\"s2\">&quot;url&quot;</span><span class=\"p\">:</span> <span class=\"n\">embedded_document</span><span class=\"o\">.</span><span class=\"n\">url</span><span class=\"p\">,</span>\n",
       "                <span class=\"s2\">&quot;text&quot;</span><span class=\"p\">:</span> <span class=\"n\">chunk</span><span class=\"o\">.</span><span class=\"n\">text</span><span class=\"p\">,</span>\n",
       "                <span class=\"s2\">&quot;embedding&quot;</span><span class=\"p\">:</span> <span class=\"n\">chunk</span><span class=\"o\">.</span><span class=\"n\">embedding</span><span class=\"p\">,</span>\n",
       "                <span class=\"s2\">&quot;metadata&quot;</span><span class=\"p\">:</span> <span class=\"n\">chunk</span><span class=\"o\">.</span><span class=\"n\">metadata</span><span class=\"p\">,</span>\n",
       "            <span class=\"p\">}</span>\n",
       "        <span class=\"p\">)</span>\n",
       "    <span class=\"n\">data_set</span> <span class=\"o\">=</span> <span class=\"n\">pd</span><span class=\"o\">.</span><span class=\"n\">DataFrame</span><span class=\"p\">(</span><span class=\"n\">rows</span><span class=\"p\">)</span>\n",
       "    <span class=\"c1\"># we create an index using the document_id and chunk_index</span>\n",
       "    <span class=\"n\">data_set</span><span class=\"o\">.</span><span class=\"n\">set_index</span><span class=\"p\">([</span><span class=\"s2\">&quot;document_id&quot;</span><span class=\"p\">,</span> <span class=\"s2\">&quot;chunk_index&quot;</span><span class=\"p\">],</span> <span class=\"n\">inplace</span><span class=\"o\">=</span><span class=\"kc\">True</span><span class=\"p\">)</span>\n",
       "    <span class=\"k\">return</span> <span class=\"n\">data_set</span>\n",
       "</pre></div>\n"
      ],
      "text/latex": [
       "\\begin{Verbatim}[commandchars=\\\\\\{\\}]\n",
       "\\PY{k+kn}{from} \\PY{n+nn}{typing} \\PY{k+kn}{import} \\PY{n}{NamedTuple}\\PY{p}{,} \\PY{n}{Optional}\n",
       "\n",
       "\\PY{k}{class} \\PY{n+nc}{Chunk}\\PY{p}{(}\\PY{n}{NamedTuple}\\PY{p}{)}\\PY{p}{:}\n",
       "\\PY{+w}{    }\\PY{l+s+sd}{\\PYZdq{}\\PYZdq{}\\PYZdq{}Chunk of a Document \\PYZhy{} text \\PYZam{} embedding.\\PYZdq{}\\PYZdq{}\\PYZdq{}}\n",
       "    \\PY{n}{index}\\PY{p}{:} \\PY{n+nb}{int}\n",
       "    \\PY{n}{document\\PYZus{}id}\\PY{p}{:} \\PY{n+nb}{str}\n",
       "    \\PY{n}{text}\\PY{p}{:} \\PY{n+nb}{str}\n",
       "    \\PY{n}{embedding}\\PY{p}{:} \\PY{n}{Optional}\\PY{p}{[}\\PY{n+nb}{list}\\PY{p}{[}\\PY{n+nb}{float}\\PY{p}{]}\\PY{p}{]}\n",
       "    \\PY{n}{metadata}\\PY{p}{:} \\PY{n}{Optional}\\PY{p}{[}\\PY{n+nb}{dict}\\PY{p}{[}\\PY{n+nb}{str}\\PY{p}{,} \\PY{n+nb}{str}\\PY{p}{]}\\PY{p}{]}\n",
       "    \n",
       "    \\PY{k}{def} \\PY{n+nf}{add\\PYZus{}embedding}\\PY{p}{(}\\PY{n+nb+bp}{self}\\PY{p}{,} \\PY{n}{embedding}\\PY{p}{:} \\PY{n+nb}{list}\\PY{p}{[}\\PY{n+nb}{float}\\PY{p}{]}\\PY{p}{)} \\PY{o}{\\PYZhy{}}\\PY{o}{\\PYZgt{}} \\PY{l+s+s2}{\\PYZdq{}}\\PY{l+s+s2}{Chunk}\\PY{l+s+s2}{\\PYZdq{}}\\PY{p}{:}\n",
       "\\PY{+w}{        }\\PY{l+s+sd}{\\PYZdq{}\\PYZdq{}\\PYZdq{}Required to update chunk with embeddings\\PYZdq{}\\PYZdq{}\\PYZdq{}}\n",
       "        \\PY{k}{return} \\PY{n}{Chunk}\\PY{p}{(}\\PY{n+nb+bp}{self}\\PY{o}{.}\\PY{n}{index}\\PY{p}{,} \\PY{n+nb+bp}{self}\\PY{o}{.}\\PY{n}{document\\PYZus{}id}\\PY{p}{,} \\PY{n+nb+bp}{self}\\PY{o}{.}\\PY{n}{text}\\PY{p}{,} \\PY{n}{embedding}\\PY{p}{,} \\PY{n+nb+bp}{self}\\PY{o}{.}\\PY{n}{metadata}\\PY{p}{)}\n",
       "    \n",
       "    \\PY{k}{def} \\PY{n+nf}{add\\PYZus{}metadata}\\PY{p}{(}\\PY{n+nb+bp}{self}\\PY{p}{,} \\PY{n}{metadata}\\PY{p}{:} \\PY{n+nb}{dict}\\PY{p}{[}\\PY{n+nb}{str}\\PY{p}{,} \\PY{n+nb}{str}\\PY{p}{]}\\PY{p}{)} \\PY{o}{\\PYZhy{}}\\PY{o}{\\PYZgt{}} \\PY{l+s+s2}{\\PYZdq{}}\\PY{l+s+s2}{Chunk}\\PY{l+s+s2}{\\PYZdq{}}\\PY{p}{:}\n",
       "\\PY{+w}{        }\\PY{l+s+sd}{\\PYZdq{}\\PYZdq{}\\PYZdq{}Required to update chunk with metadata\\PYZdq{}\\PYZdq{}\\PYZdq{}}\n",
       "        \\PY{k}{return} \\PY{n}{Chunk}\\PY{p}{(}\\PY{n+nb+bp}{self}\\PY{o}{.}\\PY{n}{index}\\PY{p}{,} \\PY{n+nb+bp}{self}\\PY{o}{.}\\PY{n}{document\\PYZus{}id}\\PY{p}{,} \\PY{n+nb+bp}{self}\\PY{o}{.}\\PY{n}{text}\\PY{p}{,} \\PY{n+nb+bp}{self}\\PY{o}{.}\\PY{n}{embedding}\\PY{p}{,} \\PY{n}{metadata}\\PY{p}{)}\n",
       "\n",
       "\\PY{k}{class} \\PY{n+nc}{Document}\\PY{p}{(}\\PY{n}{NamedTuple}\\PY{p}{)}\\PY{p}{:}\n",
       "\\PY{+w}{    }\\PY{l+s+sd}{\\PYZdq{}\\PYZdq{}\\PYZdq{}Document containing a full raw text, along with pointers to chunks.\\PYZdq{}\\PYZdq{}\\PYZdq{}}\n",
       "    \\PY{n+nb}{id}\\PY{p}{:} \\PY{n+nb}{str}\n",
       "    \\PY{n}{url}\\PY{p}{:} \\PY{n+nb}{str}\n",
       "    \\PY{n}{raw\\PYZus{}text}\\PY{p}{:} \\PY{n+nb}{str}\n",
       "    \\PY{n}{chunks}\\PY{p}{:} \\PY{n}{Optional}\\PY{p}{[}\\PY{n+nb}{list}\\PY{p}{[}\\PY{n}{Chunk}\\PY{p}{]}\\PY{p}{]}\n",
       "    \n",
       "    \\PY{k}{def} \\PY{n+nf}{add\\PYZus{}chunks}\\PY{p}{(}\\PY{n+nb+bp}{self}\\PY{p}{,} \\PY{n}{chunks}\\PY{p}{:} \\PY{n+nb}{list}\\PY{p}{[}\\PY{n}{Chunk}\\PY{p}{]}\\PY{p}{)} \\PY{o}{\\PYZhy{}}\\PY{o}{\\PYZgt{}} \\PY{l+s+s2}{\\PYZdq{}}\\PY{l+s+s2}{Document}\\PY{l+s+s2}{\\PYZdq{}}\\PY{p}{:}\n",
       "\\PY{+w}{        }\\PY{l+s+sd}{\\PYZdq{}\\PYZdq{}\\PYZdq{}Required to update the document when Chunks are created\\PYZdq{}\\PYZdq{}\\PYZdq{}}\n",
       "        \\PY{k}{return} \\PY{n}{Document}\\PY{p}{(}\\PY{n+nb+bp}{self}\\PY{o}{.}\\PY{n}{id}\\PY{p}{,} \\PY{n+nb+bp}{self}\\PY{o}{.}\\PY{n}{url}\\PY{p}{,} \\PY{n+nb+bp}{self}\\PY{o}{.}\\PY{n}{raw\\PYZus{}text}\\PY{p}{,} \\PY{n}{chunks}\\PY{p}{)}\n",
       "\n",
       "\n",
       "\\PY{k+kn}{import} \\PY{n+nn}{requests} \n",
       "\\PY{k+kn}{import} \\PY{n+nn}{re}\n",
       "\\PY{k+kn}{import} \\PY{n+nn}{uuid}\n",
       "\n",
       "\\PY{k}{def} \\PY{n+nf}{html\\PYZus{}regex}\\PY{p}{(}\\PY{p}{)} \\PY{o}{\\PYZhy{}}\\PY{o}{\\PYZgt{}} \\PY{n+nb}{str}\\PY{p}{:}\n",
       "\\PY{+w}{    }\\PY{l+s+sd}{\\PYZdq{}\\PYZdq{}\\PYZdq{}Context dependent logic for getting the right part of the HTML document.\\PYZdq{}\\PYZdq{}\\PYZdq{}}\n",
       "    \\PY{k}{return} \\PY{l+s+sa}{r}\\PY{l+s+s1}{\\PYZsq{}}\\PY{l+s+s1}{\\PYZlt{}article role=}\\PY{l+s+s1}{\\PYZdq{}}\\PY{l+s+s1}{main}\\PY{l+s+s1}{\\PYZdq{}}\\PY{l+s+s1}{ id=}\\PY{l+s+s1}{\\PYZdq{}}\\PY{l+s+s1}{furo\\PYZhy{}main\\PYZhy{}content}\\PY{l+s+s1}{\\PYZdq{}}\\PY{l+s+s1}{\\PYZgt{}(.*?)\\PYZlt{}/article\\PYZgt{}}\\PY{l+s+s1}{\\PYZsq{}}\n",
       "\n",
       "\\PY{k}{def} \\PY{n+nf}{raw\\PYZus{}document}\\PY{p}{(}\\PY{n}{url}\\PY{p}{:} \\PY{n+nb}{str}\\PY{p}{,} \\PY{n}{html\\PYZus{}regex}\\PY{p}{:} \\PY{n+nb}{str}\\PY{p}{)} \\PY{o}{\\PYZhy{}}\\PY{o}{\\PYZgt{}} \\PY{n}{Document}\\PY{p}{:}\n",
       "\\PY{+w}{    }\\PY{l+s+sd}{\\PYZdq{}\\PYZdq{}\\PYZdq{}Loads and parses the HTML from a URL, returning the html text of interest.}\n",
       "\n",
       "\\PY{l+s+sd}{    :param url: the url to pull.}\n",
       "\\PY{l+s+sd}{    :param html\\PYZus{}regex: the regext to use to get the contents out of.}\n",
       "\\PY{l+s+sd}{    :return: sub\\PYZhy{}portion of the HTML}\n",
       "\\PY{l+s+sd}{    \\PYZdq{}\\PYZdq{}\\PYZdq{}}\n",
       "    \\PY{n}{html\\PYZus{}text} \\PY{o}{=} \\PY{n}{requests}\\PY{o}{.}\\PY{n}{get}\\PY{p}{(}\\PY{n}{url}\\PY{p}{)}\\PY{o}{.}\\PY{n}{text}\n",
       "    \\PY{n}{article} \\PY{o}{=} \\PY{n}{re}\\PY{o}{.}\\PY{n}{findall}\\PY{p}{(}\\PY{n}{html\\PYZus{}regex}\\PY{p}{,} \\PY{n}{html\\PYZus{}text}\\PY{p}{,} \\PY{n}{re}\\PY{o}{.}\\PY{n}{DOTALL}\\PY{p}{)}\n",
       "    \\PY{k}{if} \\PY{o+ow}{not} \\PY{n}{article}\\PY{p}{:}\n",
       "        \\PY{k}{raise} \\PY{n+ne}{ValueError}\\PY{p}{(}\\PY{l+s+sa}{f}\\PY{l+s+s2}{\\PYZdq{}}\\PY{l+s+s2}{No article found in }\\PY{l+s+si}{\\PYZob{}}\\PY{n}{url}\\PY{l+s+si}{\\PYZcb{}}\\PY{l+s+s2}{\\PYZdq{}}\\PY{p}{)}\n",
       "    \\PY{n}{raw\\PYZus{}text} \\PY{o}{=} \\PY{n}{article}\\PY{p}{[}\\PY{l+m+mi}{0}\\PY{p}{]}\\PY{o}{.}\\PY{n}{strip}\\PY{p}{(}\\PY{p}{)}\n",
       "    \\PY{k}{return} \\PY{n}{Document}\\PY{p}{(}\\PY{n+nb}{str}\\PY{p}{(}\\PY{n}{uuid}\\PY{o}{.}\\PY{n}{uuid4}\\PY{p}{(}\\PY{p}{)}\\PY{p}{)}\\PY{p}{,} \\PY{n}{url}\\PY{p}{,} \\PY{n}{raw\\PYZus{}text}\\PY{p}{,} \\PY{k+kc}{None}\\PY{p}{)}\n",
       "\n",
       "\n",
       "\n",
       "\\PY{k+kn}{from} \\PY{n+nn}{langchain} \\PY{k+kn}{import} \\PY{n}{text\\PYZus{}splitter}\n",
       "\n",
       "\\PY{k}{def} \\PY{n+nf}{html\\PYZus{}chunker}\\PY{p}{(}\\PY{p}{)} \\PY{o}{\\PYZhy{}}\\PY{o}{\\PYZgt{}} \\PY{n}{text\\PYZus{}splitter}\\PY{o}{.}\\PY{n}{HTMLHeaderTextSplitter}\\PY{p}{:}\n",
       "\\PY{+w}{    }\\PY{l+s+sd}{\\PYZdq{}\\PYZdq{}\\PYZdq{}Object to help split HTML into chunks\\PYZdq{}\\PYZdq{}\\PYZdq{}}\n",
       "    \\PY{n}{headers\\PYZus{}to\\PYZus{}split\\PYZus{}on} \\PY{o}{=} \\PY{p}{[}\n",
       "        \\PY{p}{(}\\PY{l+s+s2}{\\PYZdq{}}\\PY{l+s+s2}{h1}\\PY{l+s+s2}{\\PYZdq{}}\\PY{p}{,} \\PY{l+s+s2}{\\PYZdq{}}\\PY{l+s+s2}{Header 1}\\PY{l+s+s2}{\\PYZdq{}}\\PY{p}{)}\\PY{p}{,}\n",
       "        \\PY{p}{(}\\PY{l+s+s2}{\\PYZdq{}}\\PY{l+s+s2}{h2}\\PY{l+s+s2}{\\PYZdq{}}\\PY{p}{,} \\PY{l+s+s2}{\\PYZdq{}}\\PY{l+s+s2}{Header 2}\\PY{l+s+s2}{\\PYZdq{}}\\PY{p}{)}\\PY{p}{,}\n",
       "        \\PY{p}{(}\\PY{l+s+s2}{\\PYZdq{}}\\PY{l+s+s2}{h3}\\PY{l+s+s2}{\\PYZdq{}}\\PY{p}{,} \\PY{l+s+s2}{\\PYZdq{}}\\PY{l+s+s2}{Header 3}\\PY{l+s+s2}{\\PYZdq{}}\\PY{p}{)}\\PY{p}{,}\n",
       "    \\PY{p}{]}\n",
       "    \\PY{n}{html\\PYZus{}chunker} \\PY{o}{=} \\PY{n}{text\\PYZus{}splitter}\\PY{o}{.}\\PY{n}{HTMLHeaderTextSplitter}\\PY{p}{(}\n",
       "        \\PY{n}{headers\\PYZus{}to\\PYZus{}split\\PYZus{}on}\\PY{o}{=}\\PY{n}{headers\\PYZus{}to\\PYZus{}split\\PYZus{}on}\n",
       "    \\PY{p}{)}\n",
       "    \\PY{k}{return} \\PY{n}{html\\PYZus{}chunker}\n",
       "\n",
       "\\PY{k}{def} \\PY{n+nf}{text\\PYZus{}chunker}\\PY{p}{(}\n",
       "        \\PY{n}{chunk\\PYZus{}size}\\PY{p}{:} \\PY{n+nb}{int} \\PY{o}{=} \\PY{l+m+mi}{256}\\PY{p}{,} \n",
       "        \\PY{n}{chunk\\PYZus{}overlap}\\PY{p}{:} \\PY{n+nb}{int} \\PY{o}{=} \\PY{l+m+mi}{0}\\PY{p}{)} \\PY{o}{\\PYZhy{}}\\PY{o}{\\PYZgt{}} \\PY{n}{text\\PYZus{}splitter}\\PY{o}{.}\\PY{n}{RecursiveCharacterTextSplitter}\\PY{p}{:}\n",
       "\\PY{+w}{    }\\PY{l+s+sd}{\\PYZdq{}\\PYZdq{}\\PYZdq{}Object to further split chunks\\PYZdq{}\\PYZdq{}\\PYZdq{}}\n",
       "    \\PY{k}{return} \\PY{n}{text\\PYZus{}splitter}\\PY{o}{.}\\PY{n}{RecursiveCharacterTextSplitter}\\PY{p}{(}\n",
       "        \\PY{n}{chunk\\PYZus{}size}\\PY{o}{=}\\PY{n}{chunk\\PYZus{}size}\\PY{p}{,} \\PY{n}{chunk\\PYZus{}overlap}\\PY{o}{=}\\PY{n}{chunk\\PYZus{}overlap}\n",
       "    \\PY{p}{)}\n",
       "\n",
       "\\PY{k}{def} \\PY{n+nf}{chunked\\PYZus{}document}\\PY{p}{(}\n",
       "    \\PY{n}{raw\\PYZus{}document}\\PY{p}{:} \\PY{n}{Document}\\PY{p}{,}\n",
       "    \\PY{n}{html\\PYZus{}chunker}\\PY{p}{:} \\PY{n}{text\\PYZus{}splitter}\\PY{o}{.}\\PY{n}{HTMLHeaderTextSplitter}\\PY{p}{,}\n",
       "    \\PY{n}{text\\PYZus{}chunker}\\PY{p}{:} \\PY{n}{text\\PYZus{}splitter}\\PY{o}{.}\\PY{n}{RecursiveCharacterTextSplitter}\\PY{p}{,}\n",
       "\\PY{p}{)} \\PY{o}{\\PYZhy{}}\\PY{o}{\\PYZgt{}} \\PY{n}{Document}\\PY{p}{:}\n",
       "\\PY{+w}{    }\\PY{l+s+sd}{\\PYZdq{}\\PYZdq{}\\PYZdq{}This function takes in HTML, chunks the HTML, and then chunks it into text chunks.\\PYZdq{}\\PYZdq{}\\PYZdq{}}\n",
       "    \\PY{n}{header\\PYZus{}splits} \\PY{o}{=} \\PY{n}{html\\PYZus{}chunker}\\PY{o}{.}\\PY{n}{split\\PYZus{}text}\\PY{p}{(}\\PY{n}{raw\\PYZus{}document}\\PY{o}{.}\\PY{n}{raw\\PYZus{}text}\\PY{p}{)}\n",
       "    \\PY{n}{text\\PYZus{}chunks} \\PY{o}{=} \\PY{n}{text\\PYZus{}chunker}\\PY{o}{.}\\PY{n}{split\\PYZus{}documents}\\PY{p}{(}\\PY{n}{header\\PYZus{}splits}\\PY{p}{)}\n",
       "    \\PY{n}{chunks} \\PY{o}{=} \\PY{p}{[}\\PY{p}{]}\n",
       "    \\PY{k}{for} \\PY{n}{i}\\PY{p}{,} \\PY{n}{text\\PYZus{}chunk} \\PY{o+ow}{in} \\PY{n+nb}{enumerate}\\PY{p}{(}\\PY{n}{text\\PYZus{}chunks}\\PY{p}{)}\\PY{p}{:}\n",
       "        \\PY{n}{chunks}\\PY{o}{.}\\PY{n}{append}\\PY{p}{(}\n",
       "            \\PY{n}{Chunk}\\PY{p}{(}\n",
       "                \\PY{n}{index}\\PY{o}{=}\\PY{n}{i}\\PY{p}{,}\n",
       "                \\PY{n}{document\\PYZus{}id}\\PY{o}{=}\\PY{n}{raw\\PYZus{}document}\\PY{o}{.}\\PY{n}{id}\\PY{p}{,}\n",
       "                \\PY{n}{text}\\PY{o}{=}\\PY{n}{text\\PYZus{}chunk}\\PY{o}{.}\\PY{n}{page\\PYZus{}content}\\PY{p}{,}\n",
       "                \\PY{n}{embedding}\\PY{o}{=}\\PY{k+kc}{None}\\PY{p}{,}\n",
       "                \\PY{n}{metadata}\\PY{o}{=}\\PY{n}{text\\PYZus{}chunk}\\PY{o}{.}\\PY{n}{metadata}\\PY{p}{,}\n",
       "            \\PY{p}{)}\n",
       "        \\PY{p}{)}\n",
       "    \\PY{c+c1}{\\PYZsh{} create new raw\\PYZus{}document object}\n",
       "    \\PY{n}{raw\\PYZus{}document} \\PY{o}{=} \\PY{n}{raw\\PYZus{}document}\\PY{o}{.}\\PY{n}{add\\PYZus{}chunks}\\PY{p}{(}\\PY{n}{chunks}\\PY{p}{)}\n",
       "    \\PY{k}{return} \\PY{n}{raw\\PYZus{}document}\n",
       "\n",
       "\n",
       "\\PY{k+kn}{import} \\PY{n+nn}{openai}\n",
       "\n",
       "\\PY{k}{def} \\PY{n+nf}{client}\\PY{p}{(}\\PY{p}{)} \\PY{o}{\\PYZhy{}}\\PY{o}{\\PYZgt{}} \\PY{n}{openai}\\PY{o}{.}\\PY{n}{OpenAI}\\PY{p}{:}\n",
       "    \\PY{k}{return} \\PY{n}{openai}\\PY{o}{.}\\PY{n}{OpenAI}\\PY{p}{(}\\PY{p}{)}\n",
       "\n",
       "\\PY{k}{def} \\PY{n+nf}{embedded\\PYZus{}document}\\PY{p}{(}\n",
       "    \\PY{n}{chunked\\PYZus{}document}\\PY{p}{:} \\PY{n}{Document}\\PY{p}{,}\n",
       "    \\PY{n}{client}\\PY{p}{:} \\PY{n}{openai}\\PY{o}{.}\\PY{n}{OpenAI}\\PY{p}{,}\n",
       "\\PY{p}{)} \\PY{o}{\\PYZhy{}}\\PY{o}{\\PYZgt{}} \\PY{n}{Document}\\PY{p}{:}\n",
       "\\PY{+w}{    }\\PY{l+s+sd}{\\PYZdq{}\\PYZdq{}\\PYZdq{}This function takes in a list of documents and outputs a list of documents with embeddings.}\n",
       "\\PY{l+s+sd}{    \\PYZdq{}\\PYZdq{}\\PYZdq{}}\n",
       "    \\PY{k}{for} \\PY{n}{idx}\\PY{p}{,} \\PY{n}{chunk} \\PY{o+ow}{in} \\PY{n+nb}{enumerate}\\PY{p}{(}\\PY{n}{chunked\\PYZus{}document}\\PY{o}{.}\\PY{n}{chunks}\\PY{p}{)}\\PY{p}{:} \n",
       "        \\PY{n}{response} \\PY{o}{=} \\PY{n}{client}\\PY{o}{.}\\PY{n}{embeddings}\\PY{o}{.}\\PY{n}{create}\\PY{p}{(}\\PY{n+nb}{input}\\PY{o}{=}\\PY{n}{chunk}\\PY{o}{.}\\PY{n}{text}\\PY{p}{,} \n",
       "                                            \\PY{n}{model}\\PY{o}{=}\\PY{l+s+s2}{\\PYZdq{}}\\PY{l+s+s2}{text\\PYZhy{}embedding\\PYZhy{}3\\PYZhy{}small}\\PY{l+s+s2}{\\PYZdq{}}\\PY{p}{)}\n",
       "        \\PY{n}{chunk} \\PY{o}{=} \\PY{n}{chunk}\\PY{o}{.}\\PY{n}{add\\PYZus{}embedding}\\PY{p}{(}\\PY{n}{response}\\PY{o}{.}\\PY{n}{data}\\PY{p}{[}\\PY{l+m+mi}{0}\\PY{p}{]}\\PY{o}{.}\\PY{n}{embedding}\\PY{p}{)}\n",
       "        \\PY{c+c1}{\\PYZsh{} mutate existing document }\n",
       "        \\PY{n}{chunked\\PYZus{}document}\\PY{o}{.}\\PY{n}{chunks}\\PY{p}{[}\\PY{n}{idx}\\PY{p}{]} \\PY{o}{=} \\PY{n}{chunk}\n",
       "    \\PY{k}{return} \\PY{n}{chunked\\PYZus{}document}\n",
       "\n",
       "\n",
       "\\PY{k+kn}{import} \\PY{n+nn}{pandas} \\PY{k}{as} \\PY{n+nn}{pd}\n",
       "\n",
       "\\PY{k}{def} \\PY{n+nf}{store}\\PY{p}{(}\n",
       "    \\PY{n}{embedded\\PYZus{}document}\\PY{p}{:} \\PY{n}{Document}\\PY{p}{,}\n",
       "\\PY{p}{)} \\PY{o}{\\PYZhy{}}\\PY{o}{\\PYZgt{}} \\PY{n}{pd}\\PY{o}{.}\\PY{n}{DataFrame}\\PY{p}{:}\n",
       "\\PY{+w}{    }\\PY{l+s+sd}{\\PYZdq{}\\PYZdq{}\\PYZdq{}Function to index \\PYZam{} store the document. Here we just put it into a pandas dataframe.\\PYZdq{}\\PYZdq{}\\PYZdq{}}\n",
       "    \\PY{c+c1}{\\PYZsh{} make a pandas dataframe from the document}\n",
       "    \\PY{c+c1}{\\PYZsh{} want a column for the document id, the chunk index, the text, the embedding, and the metadata}\n",
       "    \\PY{n}{rows} \\PY{o}{=} \\PY{p}{[}\\PY{p}{]}\n",
       "    \\PY{k}{for} \\PY{n}{chunk} \\PY{o+ow}{in} \\PY{n}{embedded\\PYZus{}document}\\PY{o}{.}\\PY{n}{chunks}\\PY{p}{:}\n",
       "        \\PY{c+c1}{\\PYZsh{} need to }\n",
       "        \\PY{n}{rows}\\PY{o}{.}\\PY{n}{append}\\PY{p}{(}\n",
       "            \\PY{p}{\\PYZob{}}\n",
       "                \\PY{l+s+s2}{\\PYZdq{}}\\PY{l+s+s2}{document\\PYZus{}id}\\PY{l+s+s2}{\\PYZdq{}}\\PY{p}{:} \\PY{n}{embedded\\PYZus{}document}\\PY{o}{.}\\PY{n}{id}\\PY{p}{,}\n",
       "                \\PY{l+s+s2}{\\PYZdq{}}\\PY{l+s+s2}{chunk\\PYZus{}index}\\PY{l+s+s2}{\\PYZdq{}}\\PY{p}{:} \\PY{n}{chunk}\\PY{o}{.}\\PY{n}{index}\\PY{p}{,}\n",
       "                \\PY{l+s+s2}{\\PYZdq{}}\\PY{l+s+s2}{url}\\PY{l+s+s2}{\\PYZdq{}}\\PY{p}{:} \\PY{n}{embedded\\PYZus{}document}\\PY{o}{.}\\PY{n}{url}\\PY{p}{,}\n",
       "                \\PY{l+s+s2}{\\PYZdq{}}\\PY{l+s+s2}{text}\\PY{l+s+s2}{\\PYZdq{}}\\PY{p}{:} \\PY{n}{chunk}\\PY{o}{.}\\PY{n}{text}\\PY{p}{,}\n",
       "                \\PY{l+s+s2}{\\PYZdq{}}\\PY{l+s+s2}{embedding}\\PY{l+s+s2}{\\PYZdq{}}\\PY{p}{:} \\PY{n}{chunk}\\PY{o}{.}\\PY{n}{embedding}\\PY{p}{,}\n",
       "                \\PY{l+s+s2}{\\PYZdq{}}\\PY{l+s+s2}{metadata}\\PY{l+s+s2}{\\PYZdq{}}\\PY{p}{:} \\PY{n}{chunk}\\PY{o}{.}\\PY{n}{metadata}\\PY{p}{,}\n",
       "            \\PY{p}{\\PYZcb{}}\n",
       "        \\PY{p}{)}\n",
       "    \\PY{n}{data\\PYZus{}set} \\PY{o}{=} \\PY{n}{pd}\\PY{o}{.}\\PY{n}{DataFrame}\\PY{p}{(}\\PY{n}{rows}\\PY{p}{)}\n",
       "    \\PY{c+c1}{\\PYZsh{} we create an index using the document\\PYZus{}id and chunk\\PYZus{}index}\n",
       "    \\PY{n}{data\\PYZus{}set}\\PY{o}{.}\\PY{n}{set\\PYZus{}index}\\PY{p}{(}\\PY{p}{[}\\PY{l+s+s2}{\\PYZdq{}}\\PY{l+s+s2}{document\\PYZus{}id}\\PY{l+s+s2}{\\PYZdq{}}\\PY{p}{,} \\PY{l+s+s2}{\\PYZdq{}}\\PY{l+s+s2}{chunk\\PYZus{}index}\\PY{l+s+s2}{\\PYZdq{}}\\PY{p}{]}\\PY{p}{,} \\PY{n}{inplace}\\PY{o}{=}\\PY{k+kc}{True}\\PY{p}{)}\n",
       "    \\PY{k}{return} \\PY{n}{data\\PYZus{}set}\n",
       "\\end{Verbatim}\n"
      ],
      "text/plain": [
       "\n",
       "from typing import NamedTuple, Optional\n",
       "\n",
       "class Chunk(NamedTuple):\n",
       "    \"\"\"Chunk of a Document - text & embedding.\"\"\"\n",
       "    index: int\n",
       "    document_id: str\n",
       "    text: str\n",
       "    embedding: Optional[list[float]]\n",
       "    metadata: Optional[dict[str, str]]\n",
       "    \n",
       "    def add_embedding(self, embedding: list[float]) -> \"Chunk\":\n",
       "        \"\"\"Required to update chunk with embeddings\"\"\"\n",
       "        return Chunk(self.index, self.document_id, self.text, embedding, self.metadata)\n",
       "    \n",
       "    def add_metadata(self, metadata: dict[str, str]) -> \"Chunk\":\n",
       "        \"\"\"Required to update chunk with metadata\"\"\"\n",
       "        return Chunk(self.index, self.document_id, self.text, self.embedding, metadata)\n",
       "\n",
       "class Document(NamedTuple):\n",
       "    \"\"\"Document containing a full raw text, along with pointers to chunks.\"\"\"\n",
       "    id: str\n",
       "    url: str\n",
       "    raw_text: str\n",
       "    chunks: Optional[list[Chunk]]\n",
       "    \n",
       "    def add_chunks(self, chunks: list[Chunk]) -> \"Document\":\n",
       "        \"\"\"Required to update the document when Chunks are created\"\"\"\n",
       "        return Document(self.id, self.url, self.raw_text, chunks)\n",
       "\n",
       "\n",
       "import requests \n",
       "import re\n",
       "import uuid\n",
       "\n",
       "def html_regex() -> str:\n",
       "    \"\"\"Context dependent logic for getting the right part of the HTML document.\"\"\"\n",
       "    return r'<article role=\"main\" id=\"furo-main-content\">(.*?)</article>'\n",
       "\n",
       "def raw_document(url: str, html_regex: str) -> Document:\n",
       "    \"\"\"Loads and parses the HTML from a URL, returning the html text of interest.\n",
       "\n",
       "    :param url: the url to pull.\n",
       "    :param html_regex: the regext to use to get the contents out of.\n",
       "    :return: sub-portion of the HTML\n",
       "    \"\"\"\n",
       "    html_text = requests.get(url).text\n",
       "    article = re.findall(html_regex, html_text, re.DOTALL)\n",
       "    if not article:\n",
       "        raise ValueError(f\"No article found in {url}\")\n",
       "    raw_text = article[0].strip()\n",
       "    return Document(str(uuid.uuid4()), url, raw_text, None)\n",
       "\n",
       "\n",
       "\n",
       "from langchain import text_splitter\n",
       "\n",
       "def html_chunker() -> text_splitter.HTMLHeaderTextSplitter:\n",
       "    \"\"\"Object to help split HTML into chunks\"\"\"\n",
       "    headers_to_split_on = [\n",
       "        (\"h1\", \"Header 1\"),\n",
       "        (\"h2\", \"Header 2\"),\n",
       "        (\"h3\", \"Header 3\"),\n",
       "    ]\n",
       "    html_chunker = text_splitter.HTMLHeaderTextSplitter(\n",
       "        headers_to_split_on=headers_to_split_on\n",
       "    )\n",
       "    return html_chunker\n",
       "\n",
       "def text_chunker(\n",
       "        chunk_size: int = 256, \n",
       "        chunk_overlap: int = 0) -> text_splitter.RecursiveCharacterTextSplitter:\n",
       "    \"\"\"Object to further split chunks\"\"\"\n",
       "    return text_splitter.RecursiveCharacterTextSplitter(\n",
       "        chunk_size=chunk_size, chunk_overlap=chunk_overlap\n",
       "    )\n",
       "\n",
       "def chunked_document(\n",
       "    raw_document: Document,\n",
       "    html_chunker: text_splitter.HTMLHeaderTextSplitter,\n",
       "    text_chunker: text_splitter.RecursiveCharacterTextSplitter,\n",
       ") -> Document:\n",
       "    \"\"\"This function takes in HTML, chunks the HTML, and then chunks it into text chunks.\"\"\"\n",
       "    header_splits = html_chunker.split_text(raw_document.raw_text)\n",
       "    text_chunks = text_chunker.split_documents(header_splits)\n",
       "    chunks = []\n",
       "    for i, text_chunk in enumerate(text_chunks):\n",
       "        chunks.append(\n",
       "            Chunk(\n",
       "                index=i,\n",
       "                document_id=raw_document.id,\n",
       "                text=text_chunk.page_content,\n",
       "                embedding=None,\n",
       "                metadata=text_chunk.metadata,\n",
       "            )\n",
       "        )\n",
       "    # create new raw_document object\n",
       "    raw_document = raw_document.add_chunks(chunks)\n",
       "    return raw_document\n",
       "\n",
       "\n",
       "import openai\n",
       "\n",
       "def client() -> openai.OpenAI:\n",
       "    return openai.OpenAI()\n",
       "\n",
       "def embedded_document(\n",
       "    chunked_document: Document,\n",
       "    client: openai.OpenAI,\n",
       ") -> Document:\n",
       "    \"\"\"This function takes in a list of documents and outputs a list of documents with embeddings.\n",
       "    \"\"\"\n",
       "    for idx, chunk in enumerate(chunked_document.chunks): \n",
       "        response = client.embeddings.create(input=chunk.text, \n",
       "                                            model=\"text-embedding-3-small\")\n",
       "        chunk = chunk.add_embedding(response.data[0].embedding)\n",
       "        # mutate existing document \n",
       "        chunked_document.chunks[idx] = chunk\n",
       "    return chunked_document\n",
       "\n",
       "\n",
       "import pandas as pd\n",
       "\n",
       "def store(\n",
       "    embedded_document: Document,\n",
       ") -> pd.DataFrame:\n",
       "    \"\"\"Function to index & store the document. Here we just put it into a pandas dataframe.\"\"\"\n",
       "    # make a pandas dataframe from the document\n",
       "    # want a column for the document id, the chunk index, the text, the embedding, and the metadata\n",
       "    rows = []\n",
       "    for chunk in embedded_document.chunks:\n",
       "        # need to \n",
       "        rows.append(\n",
       "            {\n",
       "                \"document_id\": embedded_document.id,\n",
       "                \"chunk_index\": chunk.index,\n",
       "                \"url\": embedded_document.url,\n",
       "                \"text\": chunk.text,\n",
       "                \"embedding\": chunk.embedding,\n",
       "                \"metadata\": chunk.metadata,\n",
       "            }\n",
       "        )\n",
       "    data_set = pd.DataFrame(rows)\n",
       "    # we create an index using the document_id and chunk_index\n",
       "    data_set.set_index([\"document_id\", \"chunk_index\"], inplace=True)\n",
       "    return data_set"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%print_module doc_pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "02904414",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-03T22:36:24.882367Z",
     "start_time": "2024-05-03T22:36:24.878Z"
    },
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reset doc_pipeline\n"
     ]
    }
   ],
   "source": [
    "# use %reset_module NAME to reset the module.\n",
    "# %reset_module doc_pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5368bbb6",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
